About this fork

This fork was built for Ontario Tech University's CSCI 6720 course, extending the upstream MINERS benchmark with an ICL NER pipeline and a Dense Passage Retrieval (DPR) pipeline. Compared against upstream/main, the fork adds:

• ICL for NER: new icl_NER.py (358 lines) and icl_open_end_gen.py (244 lines) scripts, backed by a new MasakhaNERDataset class added to utils.py, extending the original ICL classification code to sequence-labeling on the Masakhaner NER dataset.
• A DPR pipeline: a new DPR/ directory (chunk_dataset.py, encode_corpus.sh, encoding.sh, merge_embeddings.py, merge.sh, json_to_tsv.py, search_eval.sh, search_f.sh, corpus_change.py, job.sh) that chunks and encodes a corpus (e.g., the MIRACL corpus) and runs retrieval search/evaluation, illustrated by the new assets/DPR.png and assets/pipeline_extended.png diagrams.
• New dataset support: OpenSubtitlesDataset and XLSumDataset classes added to utils.py (OpenSubtitlesDataset is wired into bitext.py for the opensub option), plus a new English-Hinglish code-switching MT dataset (train.txt/dev.txt/test.txt) used by a bitext dataset class in utils.py.
• Recorded bitext-mining results: 67 new JSON result files under outputs/save_bitext/ (across nusax, bucc, opensub, nollysenti, for the LaBSE and Conan-embedding-v1 encoders), aggregated in a new outputs/save_bitext/bitext_logs.ipynb notebook. For example, outputs/save_bitext/nusax/sentence-transformers/LaBSE/seed_42/eval_eng_bbc_5.json reports "acc": 0.748 and "f1": 0.6954652236652237.

Credit to the original MINERS project and paper (Winata, Zhang, and Adelani, EMNLP 2024 Findings) at gentaiscool/miners, on which this fork is based.

Extending the MINERS Framework

This fork was developed for Ontario Tech University's CSCI 6720 Group Project, focusing on extending MINERS Framework's ICL classification and Deep Passage Retrieval tasks.

Group Members:

• Alexie Linardatos(alexie.linardatos@ontariotechu.net)
• Madhav Goyani(madhav.goyani@ontariotechu.net)
• Zikun Fu(zikun.fu@ontariotechu.net)

A 3 minute video summary on this project can be found here: Link

The final report can be found here: Link

Environment Setup

pip install -r requirements.txt

Microsoft Visual C++ 14.0 or greater is required to run the project. You can download it from Microsoft's website.

Experiment Logs

Full Experiment logs can be accessed here.

Running Experiments

ICL Classification

python icl_NER.py --dataset {dataset} --seed 42 --model_checkpoint {model} --gen_model_checkpoint {gen_model_checkpoint}  --cuda --load_in_8bit --k {k}
python icl_NER.py --dataset masakhaner --seed 42 --model_checkpoint sentence-transformers/LaBSE --gen_model_checkpoint meta-llama/Meta-Llama-3.1-8B-Instruct --cuda --load_in_8bit --k 2

Dense Passage Retrieval

Download the corpus:

!wget https://huggingface.co/datasets/miracl/miracl-corpus/resolve/main/miracl-corpus-v1.0-bn/docs-0.jsonl.gz

Chunk and Encode the downloaded corpus:

python chunk_dataset.py
bash encode_corpus.sh

Run the evaluation:

bash search_eval.sh

Models Support

All models used for the experiments are listed below:

Encoder LMs and APIs

Open-source LMs:

• sentence-transformers/LaBSE
• intfloat/multilingual-e5-base
• FacebookAI/xlm-roberta-base

Generative LMs:

• Gemma 2 Instruct google/gemma-2-9b-it
• Llama 3 8B Instruct meta-llama/Meta-Llama-3.1-8B-Instruct

Credits

• OpenSub bitext mining dataset
• Masakhaner NER dataset
• MIRACL dataset
• Framework code based on the MINERS paper:

  @article{winata2024miners,
  title={MINERS: Multilingual Language Models as Semantic Retrievers},
  author={Winata, Genta Indra and Zhang, Ruochen and Adelani, David Ifeoluwa},
  journal={arXiv preprint arXiv:2406.07424},
  year={2024}
  }