Samenvatting voor Information Retrieval Exam (X_400435) voor minor Data Science aan de VU. Informatie: Lecture 1: Introduction (book chapter 1)
Lecture 2: Indexing and Boolean Retrieval (book chapters 2 and 4)
Lecture 3: What to Index? (book chapters 2 and 3)
Lecture 4: Beyond Simple Queries (bo...
Samenvatting Introduction to Information Retrieval
All for this textbook (4)
Written for
Vrije Universiteit Amsterdam (VU)
Artificial Intelligence
Information Retrieval (X_400435)
All documents for this subject (3)
Seller
Follow
simonvanrens
Reviews received
Content preview
Information retrieval 1
Is finding material of an unstructured nature that satisfies an information need from within
large collections (usually stored on computers)
First idea for an automated system was 1945 by Vannevar Bush in As We May Think
In 1960s the field of Information Retreival emerged
Evolution of IR
1960-70s: era of Boolean Retrieval
1975s: first Vector Space Model
1980s: large document database systems run by companies became available (LexisNexis,
MedLine)
1990s: FTP search and the dawn of Web search (lycos, Yahoo)
IR in 2000s
Google
- Link analysis & ranking
- Multimedia IR (image and video analysis)
- Cross-language IR
- Semantic Web Technologies (DBPedia)
IR since 2010s
Categorization and clustering, and recommendation:
- iTunes “Top Songs”
- Amazon “people who bought this also bought …”
- IBMs Watson system (business related: predict future outcomes)
- Recommendations in Netflix, spotify, youtube
IR versus DB
IR DB (databases)
Unstructured data structured
Set of keywords (loose semantics) well defined query (SQL)
Incomplete query specification, partial matching complete query specification, exact matc
Relevant items for result, errors tolerable single error results in failure
Probabilistic models deterministic models
,What is needed to build a search engine
What makes a search engine good?
Speed + User happiness
Which of following actions if fastest and slowest?
1 – main memory reference (read random byte from memory) 1
2 – Hard disk seek (read random byte from hard disk) 5
3 – SSD random read (read random byte from solid-state drive) 3
4 – Zip 1KB of data (compress 1000 bytes in memory) 2
5 – Round trip within same datacenter (send one byte to another computer in same fast
datacenter network and back) 4
6 – Send one byte from Netherlands to California and back 6
,2:
In memory, can use linked lists or variable-length arrays
Token = an instance of a sequence of
characters in some particular document that are grouped together as a useful semantic unit
for processing
Type = the class of all tokens consisting of exactly the same character sequence
Term = a (perhaps normalized) type that is included in the IR systems dictionary
Bottleneck: sorting
Sorting lot of records on disk is much too slow – in particular for hard disks but also for SSDs
but data is too large for memory, so we need an external sorting algo
, Summary
Boolean retrieval:
A simple and well-understood retrieval model
Inverted indexes:
Inverting an index demands a lot of resources
Sorting the index is the critical step
3:
Biword indexes: Index every consecutive pair of terms in the text as a phrase
So “Friends, Romans, Countrymen would generate biwrods: friends romans & romans
countrymen
So we can now process 2-word phrase queries in a straightforward manner
But with longer phrase queries could be false positives, we cannot verify that it contains
query
Problems:
False positives in answer set
In particular for phrases with frequent words like “beer of the month”
Index blow-up due to bigger dictionary
Infeasible for more than biwords
Biword indexes are therefore not the standard solution, but they can be part of a compound
strategy
The benefits of buying summaries with Stuvia:
Guaranteed quality through customer reviews
Stuvia customers have reviewed more than 700,000 summaries. This how you know that you are buying the best documents.
Quick and easy check-out
You can quickly pay through credit card or Stuvia-credit for the summaries. There is no membership needed.
Focus on what matters
Your fellow students write the study notes themselves, which is why the documents are always reliable and up-to-date. This ensures you quickly get to the core!
Frequently asked questions
What do I get when I buy this document?
You get a PDF, available immediately after your purchase. The purchased document is accessible anytime, anywhere and indefinitely through your profile.
Satisfaction guarantee: how does it work?
Our satisfaction guarantee ensures that you always find a study document that suits you well. You fill out a form, and our customer service team takes care of the rest.
Who am I buying these notes from?
Stuvia is a marketplace, so you are not buying this document from us, but from seller simonvanrens. Stuvia facilitates payment to the seller.
Will I be stuck with a subscription?
No, you only buy these notes for $12.32. You're not tied to anything after your purchase.
