Семантический поиск через Embed Jobs

Урок 3 из 114 курса «Cohere: LLM University»: официальный курс Cohere LLM University (Кохир) на русском языке. Этот урок бесплатный.

Семантический поиск с Cohere Embed Jobs

# TODO: upgrade to "cohere>5"
! pip install "cohere<5" hnswlib -q
import time
import cohere
import hnswlib
co = cohere.Client('COHERE_API_KEY')

Шаг 1: Загрузите набор данных

# Загрузка набора данных для заданий embed
# Этот пример набора данных содержит статьи Википедии по следующим темам: Youtube, США, Великобритания, Елизавета II, Википедия, Чемпионат мира по футболу 2022, Microsoft Office, Индия, Криштиану Роналду, Клеопатра, Instagram, Facebook и Украина

dataset_file_path = "data/embed_jobs_sample_data.jsonl" # Полный путь - https://raw.githubusercontent.com/cohere-ai/cohere-developer-experience/main/notebooks/data/embed_jobs_sample_data.jsonl

ds=co.create_dataset(
	name='sample_file',
	data=open(dataset_file_path, 'rb'),
	keep_fields = ['id','wiki_id'],
	dataset_type="embed-input"
	)
uploading file, starting validation...
sample-file-hca4x0 was uploaded
...

print(ds.await_validation())

Шаг 2: Создайте embeddings через endpoint Embed Jobs от Cohere

# Набор данных загружен, создайте задание embed и укажите тип ввода как "search document", так как это будет храниться в вашей базе данных Pinecone.
job = co.create_embed_job(
    dataset_id=ds.id,
    input_type='search_document' ,
    model='embed-english-v3.0', 
    embeddings_types=['float'])

job.wait() # опрашивать сервер, пока задание не будет завершено
...
...

print(job)

Шаг 3: Загрузите и подготовьте embeddings

# Сохранить результат задания
embeddings_file_path = 'embed_jobs_output.csv'
output_dataset=co.get_dataset(job.output.id)
output_dataset.save(filepath=embeddings_file_path, format="csv")
# Добавить результаты
embeddings=[]
texts=[]
for record in output_dataset:
  embeddings.append(record['embeddings']['float'])
  texts.append(record['text'])

Шаг 4: Инициализируйте индекс Hnwslib и добавьте embeddings

# Создать индекс hnsw
index = hnswlib.Index(space='ip', dim=1024)
index.init_index(max_elements=len(embeddings), ef_construction=512, M=64)
index.add_items(embeddings,list(range(len(embeddings))))

Шаг 5: Запросите индекс и переранжируйте результаты

# Запрос к базе данных
query = "What was the first youtube video about?"

# Преобразовать запрос в embeddings
query_emb=co.embed(
    texts=[query], model="embed-english-v3.0", input_type="search_query"
        ).embeddings

# Получить первоначальные результаты из вашей векторной базы данных
doc_index = index.knn_query(query_emb, k=10)[0][0]

# Из doc_index получить текст из каждого индекса, а затем передать текст в rerank
docs_to_rerank = []
for index in doc_index:
  docs_to_rerank.append(texts[index])

final_result = co.rerank(
    query= query,
    documents=docs_to_rerank,
    model="rerank-english-v2.0",
    top_n=3)

Шаг 6: Отобразите результаты

# Вывод результатов
for idx, r in enumerate(final_result):
  print(f"Ранг документа: {idx + 1}, Индекс документа: {r.index}")
  print(f"Документ: {r.document['text']}")
  print(f"Оценка релевантности: {r.relevance_score:.5f}")
  print("\n")
Document Rank: 1, Document Index: 0
Document: YouTube began as a venture capital–funded technology startup. Between November 2005 and April 2006, the company raised money from various investors, with Sequoia Capital, $11.5 million, and Artis Capital Management, $8 million, being the largest two. YouTube's early headquarters were situated above a pizzeria and a Japanese restaurant in San Mateo, California. In February 2005, the company activated codice_1. The first video was uploaded April 23, 2005. Titled "Me at the zoo", it shows co-founder Jawed Karim at the San Diego Zoo and can still be viewed on the site. In May, the company launched a public beta and by November, a Nike ad featuring Ronaldinho became the first video to reach one million total views. The site launched officially on December 15, 2005, by which time the site was receiving 8 million views a day. Clips at the time were limited to 100 megabytes, as little as 30 seconds of footage.
Relevance Score: 0.94815


Document Rank: 2, Document Index: 1
Document: Karim said the inspiration for YouTube first came from the Super Bowl XXXVIII halftime show controversy when Janet Jackson's breast was briefly exposed by Justin Timberlake during the halftime show. Karim could not easily find video clips of the incident and the 2004 Indian Ocean Tsunami online, which led to the idea of a video-sharing site. Hurley and Chen said that the original idea for YouTube was a video version of an online dating service, and had been influenced by the website Hot or Not. They created posts on Craigslist asking attractive women to upload videos of themselves to YouTube in exchange for a $100 reward. Difficulty in finding enough dating videos led to a change of plans, with the site's founders deciding to accept uploads of any video.
Relevance Score: 0.91626


Document Rank: 3, Document Index: 2
Document: YouTube was not the first video-sharing site on the Internet; Vimeo was launched in November 2004, though that site remained a side project of its developers from CollegeHumor at the time and did not grow much, either. The week of YouTube's launch, NBC-Universal's "Saturday Night Live" ran a skit "Lazy Sunday" by The Lonely Island. Besides helping to bolster ratings and long-term viewership for "Saturday Night Live", "Lazy Sunday"'s status as an early viral video helped establish YouTube as an important website. Unofficial uploads of the skit to YouTube drew in more than five million collective views by February 2006 before they were removed when NBCUniversal requested it two months later based on copyright concerns. Despite eventually being taken down, these duplicate uploads of the skit helped popularize YouTube's reach and led to the upload of more third-party content. The site grew rapidly; in July 2006, the company announced that more than 65,000 new videos were being uploaded every day and that the site was receiving 100 million video views per day.
Relevance Score: 0.90665

Полезные гиды