Better feature extraction algorithm #1

New Issue

hjp · 2019-09-01T15:22:41+02:00

hjp commented

2019-09-01 15:22:41 +02:00

Avoid overlapping evidence by:

put evidence in a list.
find all substrings
find spam-probability of all substrings·
order by prob, length
for each substring:
    if in evidence:
        keep.
        use it split each member of evidence list into new members
    else:
        skip

Avoid overlapping evidence by: ``` put evidence in a list. find all substrings find spam-probability of all substrings· order by prob, length for each substring: if in evidence: keep. use it split each member of evidence list into new members else: skip ```

hjp commented

2019-09-01 15:26:08 +02:00

Downside: We may need to fetch a lot more evidence from the database.

Maybe do it in batches? First get the first 100, then the second, etc. until we have enough evidence. Since we only use the first 20, I think there is a good chance that the first 100 will be enough.

Or maybe we are already fetching everything anyway? Need to check.

Downside: We may need to fetch a lot more evidence from the database. Maybe do it in batches? First get the first 100, then the second, etc. until we have enough evidence. Since we only use the first 20, I think there is a good chance that the first 100 will be enough. Or maybe we are already fetching everything anyway? Need to check.

hjp referenced this issue from a commit

2019-09-14 11:06:05 +02:00

Improve overlap avoidance (#1) When a feature is used, we use it to split the input string in which it was found and use the fragments for subsequent feature searches. So overlaps are impossible.

hjp commented

2019-09-14 11:07:16 +02:00

Fixed in d96d1fc

Fixed in d96d1fc

hjp closed this issue

2019-09-14 11:07:16 +02:00

Sign in to join this conversation.