MetaCommit

data/LLM_English/english_summary.jsonl

@@ -0,0 +1,19 @@
+{"type": "english_summarize", "instruction": "Summarize the main claim made in the first paragraph of the document about achieving world-class English proficiency in LLMs.", "input": "", "reasoning": "The paragraph argues that a paradigm shift from raw data volume to meticulous data curation is necessary for graduate-level reasoning and native-like grasp of English.", "output": "To build an LLM with world-class English proficiency (graduate-level reasoning, grammatical precision, deep contemplation, and colloquial pragmatics), researchers must abandon the brute-force method of unfiltered web data. Instead, a paradigm shift toward meticulously stratified data curation is required to overcome the functional brittleness of statistically fluent models."},
+{"type": "english_summarize", "instruction": "Summarize the empirical benefits of pre-training on FineWeb-Edu as reported in the document.", "input": "", "reasoning": "The document states a ~12% relative gain on MMLU (33% to 37% accuracy) and a 24% gain on ARC (46% to 57%).", "output": "Pre-training on FineWeb-Edu yields significant performance improvements: a 12% relative gain on the Massive Multitask Language Understanding (MMLU) benchmark, raising accuracy from 33% to 37%, and a 24% gain on the AI2 Reasoning Challenge (ARC), raising scores from 46% to 57%. These gains demonstrate that educational filtering directly enhances academic reasoning and factual consistency."},
+{"type": "english_summarize", "instruction": "Summarize the domain-specific subsets of the Common Corpus and their linguistic benefits.", "input": "", "reasoning": "Open Culture (885.9B tokens): classic literature, historical newspapers → teaches archaic vocabulary and classical syntax. Open Government (406.6B tokens): financial and legal documents → teaches formal, precise professional register.", "output": "The Common Corpus includes two key English subsets: (1) Open Culture with 885.9 billion tokens from classic literature, historical newspapers, and cultural heritage texts — preventing temporal myopia by teaching archaic vocabulary and classical sentence structures. (2) Open Government with 406.6 billion tokens from financial documents (Finance Commons) and legal case law (Legal Commons from Free Law Project, EUR-Lex, USPTO) — enabling the model to adopt a highly formal, precise, professional register for complex regulatory or economic subjects."},
+{"type": "english_summarize", "instruction": "Summarize the key findings about the Lang-8 dataset and its role in grammatical error detection.", "input": "", "reasoning": "Lang-8 has 2.3M rows of learner errors corrected by native speakers. Fine-tuning on cleaned versions achieves 0.91 F1 score and 90.53% accuracy for error detection. Native vs non-native error patterns differ.", "output": "The Lang-8 dataset contains over 2.3 million rows of language learner errors manually corrected by native speakers. Fine-tuning transformer models on cleaned iterations achieves an F1 score of 0.91 and 90.53% accuracy for Grammatical Error Detection. Critically, native and non-native speakers make different error types: native speakers struggle with punctuation and spelling, while non-native speakers struggle with determiners and articles. Training on Lang-8 ensures robustness against all error types."},
+{"type": "english_summarize", "instruction": "Summarize what bridging anaphora is and how the GUMBridge corpus trains LLMs to handle it.", "input": "", "reasoning": "Bridging anaphora: understanding a new entity via its relation to a previous non-identical entity (e.g., 'house' → 'door'). GUMBridge provides granular annotations across 24 genres to teach deep contextual memory and coreference resolution.", "output": "Bridging anaphora is a linguistic phenomenon where a newly introduced entity is understood solely through its relation to a previously mentioned, non-identical entity (e.g., 'There is a house. The door is red' — 'the door' is inferable as belonging to 'the house'). The GUMBridge corpus, a subset of the Georgetown University Multilayer corpus, spans 24 diverse English genres with granular annotations for bridging anaphora. Training on GUMBridge forces the LLM to develop deep contextual memory and complex coreference resolution skills, marking the dividing line between disjointed sentence generation and cohesive, logically flowing paragraphs."},
+{"type": "english_summarize", "instruction": "Summarize the GPQA and MMLU-Pro benchmarks and how they evaluate graduate-level reasoning.", "input": "", "reasoning": "GPQA contains graduate-level questions in physics, chemistry, biology designed to be unsolvable by search engines. MMLU-Pro expands from 4 to 10 answer options across 12,000 questions in 14 fields, requiring extreme confidence and precision.", "output": "GPQA (Graduate-Level Google-Proof Q&A) contains highly complex questions in physics, chemistry, and biology requiring deep subject-matter knowledge, calculation, and concept synthesis. These questions are explicitly designed to be unsolvable by simple search engines, forcing the model to rely on internalized academic logic. MMLU-Pro represents a significant escalation from the original MMLU benchmark: it includes 12,000 high-quality academic exam questions spanning 14 major fields, with answer options expanded from 4 to 10. This expanded distractor set eliminates random guessing or shallow heuristic matching, requiring extreme confidence and precision in knowledge retrieval."},
+{"type": "english_summarize", "instruction": "Summarize the GRAPE framework and its efficiency benefits for instruction tuning.", "input": "", "reasoning": "GRAPE aligns SFT data with the model's pre-trained distribution by selecting responses that match the target model's probability distribution. It achieves up to 13.8% gains and requires 1/3 the data and half the epochs to exceed baselines trained on 4.5x more data.", "output": "The GRAPE framework revolutionizes Supervised Fine-Tuning (SFT) by aligning instructional data with the model's inherent pre-trained probability distribution. For each instruction, GRAPE gathers responses from multiple sources (human annotators or larger oracle models) and selects the specific response that most closely matches the target model's pre-trained distribution. Standard SFT then runs only on these probabilistically aligned responses. In controlled experiments on Ultralnteract and OpenHermes-2.5, GRAPE-selected data yielded absolute gains up to 13.8% across benchmarks. Remarkably, using GRAPE to subsample post-training data, a smaller model (LLaMA-1-8B) required only one-third of the data and half the training epochs to exceed baselines trained on 4.5 times more data."},
+{"type": "english_summarize", "instruction": "Summarize the LiteraryTaste dataset and how it captures nuanced human preferences for writing style.", "input": "", "reasoning": "LiteraryTaste contains 100 pairs of short creative writing texts annotated by 60 individuals revealing revealed vs stated reading preferences. Fine-tuning on it achieves >75% accuracy in modeling personal preferences, allowing dynamic register shifting.", "output": "The LiteraryTaste dataset captures nuanced human preferences for creative writing style. It contains 100 pairs of short creative writing texts, annotated by 60 individuals who detailed their revealed preferences (what they actually enjoyed reading) versus stated preferences (what they claimed to like). Fine-tuning a transformer encoder on this dataset achieves over 75% accuracy in modeling revealed personal preferences. This allows the LLM to adapt its prose dynamically — shifting seamlessly from the ornate, polysyllabic elegance of Victorian literature to the punchy, minimalist clarity of modern journalism — based entirely on the user's implicit needs and preferences."},
+{"type": "english_summarize", "instruction": "Summarize the PodSarc dataset and how it complements SARC for sarcasm detection.", "input": "", "reasoning": "PodSarc provides large-scale sarcastic speech from podcasts with average 31.18 words over 9.61 seconds per segment. It captures spoken English nuances: conversational shifts, rhetorical questions, deadpan delivery — things absent from written text.", "output": "The PodSarc dataset offers a large-scale collection of sarcastic speech derived from podcasts. With an average of 31.18 words spanning 9.61 seconds per segment, this dataset provides lengthy, dialogue-heavy contexts necessary for detecting conversational shifts, rhetorical questions, and deadpan delivery. It complements SARC by capturing the nuances of spoken English translated into text — features like prosody, timing, and intonation patterns that are implicit in speech but absent from written Reddit comments. Together, these datasets ensure the LLM can detect sarcasm whether it appears in informal online writing or transcribed conversation, a critical capability for true native-level pragmatic understanding."},
+{"type": "english_summarize", "instruction": "Summarize how the MAGPIE and IDEM datasets train LLMs to understand idiomatic expressions.", "input": "", "reasoning": "MAGPIE provides 56,000+ instances of 1,756 unique English idioms in varied contexts. IDEM attaches emotion labels to nearly 10,000 idiom-containing sentences, teaching the affective weight of figurative language.", "output": "MAGPIE addresses the challenge of semantic opacity in idioms — where meaning cannot be deduced from literal word definitions — by providing over 56,000 instances of 1,756 unique English idioms presented within short, highly varied textual contexts. This exposes the model to idioms in diverse syntactic and semantic environments, preventing memorization of fixed patterns. Complementing this, the IDEM dataset attaches specific emotion labels to nearly 10,000 idiom-containing sentences, teaching the model the affective weight and emotional resonance of figurative language — for example, that 'kicked the bucket' carries different pragmatic force than 'passed away' despite similar literal reference. Together, these datasets ensure the LLM understands both the non-literal meaning and emotional register of idiomatic expressions."},
+{"type": "english_summarize", "instruction": "Summarize the Jigsaw Toxic Comment Classification dataset and its role in teaching ethical boundary adherence.", "input": "", "reasoning": "Jigsaw contains 160,000 human-annotated Wikipedia talk page comments categorized into sub-labels: severe toxicity, insults, threats, identity-based hate. It teaches how benign words can be weaponized and how profanity can be harmless in colloquial contexts.", "output": "The Jigsaw Toxic Comment Classification dataset contains 160,000 human-annotated Wikipedia talk page comments, categorized into distinct sub-labels including severe toxicity, insults, threats, and identity-based hate. This dataset serves as the primary mechanism for teaching the model nuanced content moderation: how seemingly benign words can be weaponized into abuse through context (e.g., 'interesting' said sarcastically to dismiss someone's identity), and conversely, how profanity can be used harmlessly for emphasis in colloquial contexts (e.g., 'That's f---ing awesome!' as positive exclamation). Training on Jigsaw ensures the LLM possesses ethical discernment and pragmatic awareness — knowing exactly when offensive language is unacceptable (harassment, hate speech) versus when it might be appropriate (creative writing, specific roleplay scenarios, casual conversation among consenting adults)."},
+{"type": "english_summarize", "instruction": "Summarize the document's core argument about why indiscriminate scaling fails and what succeeds instead.", "input": "", "reasoning": "Indiscriminate scaling (more parameters, more raw web data) yields diminishing returns because noisy data teaches surface patterns, not deep understanding. Success requires meticulous, multi-stage data curation with specialized datasets for syntax, reasoning, style, and pragmatics.", "output": "The document's core argument is that achieving world-class English proficiency in LLMs cannot be achieved by indiscriminately expanding context windows or increasing parameter counts over raw, noisy web data. Such brute-force approaches yield statistically fluent but functionally brittle models that fail at sub-surface complexities like sarcasm, bridging anaphora, or graduate-level reasoning. Instead, success requires a meticulous, multi-stage data curation paradigm: (1) educationally filtered pre-training for structured exposition, (2) syntactic datasets for grammatical perfection and coreference resolution, (3) Chain-of-Thought and reasoning datasets for step-by-step logic, (4) literary and philosophical corpora for elegant prose, and (5) pragmatic datasets for sarcasm, humor, idioms, and profanity. Only this carefully sequenced symphony of specialized datasets produces a model that is genuinely 'PhD-smart,' deeply thoughtful, and capable of native-like communication."},
+{"type": "english_summarize", "instruction": "Summarize the benchmarking performance of FineWeb-Edu on MMLU and other reasoning tasks.", "input": "", "reasoning": "FineWeb-Edu improves MMLU from 33% to approximately 37% and shows substantial gains on ARC and OpenBookQA.", "output": "FineWeb-Edu demonstrates substantial empirical gains on knowledge- and reasoning-intensive benchmarks. On MMLU, models pretrained on FineWeb-Edu improve from 33% to approximately 37% accuracy. It also yields significant improvements on the AI2 Reasoning Challenge (ARC) and OpenBookQA, making it one of the most effective open educational web datasets for LLM pretraining."},
+{"type": "english_summarize", "instruction": "Summarize the licensing and ethical compliance features of Common Corpus.", "input": "", "reasoning": "All components are either public domain or under permissive licenses (CC-By, CC0, MIT, Apache, BSD). Extensive provenance documentation ensures legal and ethical compliance for commercial AI deployment.", "output": "Common Corpus is assembled exclusively from uncopyrighted or permissively licensed sources. All components are either public domain or distributed under licenses such as CC-By, CC0, MIT, Apache, or BSD. The dataset includes exhaustive documentation of provenance, licensing, and curation procedures, ensuring compliance with data security regulations and intellectual property laws. This design makes Common Corpus suitable for both scientific research and commercial AI deployment under emerging AI legislation."},
+{"type": "english_summarize", "instruction": "Summarize the relationship between S2ORC, the Semantic Scholar Academic Graph (S2AG), and the overall Ai2 open data ecosystem.", "input": "", "reasoning": "S2AG provides 200M paper titles/abstracts/citations as the underlying graph. S2ORC adds structured full text for a subset. Together they form the largest open scientific literature graph.", "output": "The Semantic Scholar Academic Graph (S2AG) is a large collection of over 200 million paper titles, abstracts, citations, and other metadata from open access papers, forming the largest open scientific literature graph. S2ORC builds on S2AG by adding structured full text for 8.1 million open access papers, with annotations for citations, figures, and tables. Together, these resources provide a comprehensive, machine-readable representation of the scientific literature, enabling tasks such as citation analysis, knowledge graph construction, and scientific language model pretraining."},
+{"type": "english_summarize", "instruction": "Summarize the step-by-step instruction strategy for improving LLM dependency parsing accuracy using CoNLL-U format.", "input": "", "reasoning": "Recent research shows that requiring LLMs to predict universal part-of-speech tags before syntactic heads and dependency labels improves accuracy across 17 languages without hallucination.", "output": "A 2025 study proposes a step-by-step instruction strategy for LLM-based dependency parsing: first, predict universal part-of-speech (UPOS) tags for each token; second, predict syntactic heads and dependency labels using the UPOS tags as intermediate reasoning steps. This method uses a simplified CoNLL-U-like output format and achieves state-of-the-art accuracy on Universal Dependencies datasets across 17 languages without hallucination or data contamination. The approach highlights the effectiveness of explicit reasoning steps in LLM-based parsing, offering a scalable alternative to bracket-based methods."},
+{"type": "english_summarize", "instruction": "Summarize the empirical performance gains achieved by GRAPE on Llama3.1-8B and other models.", "input": "", "reasoning": "GRAPE achieves absolute gains up to 13.8% across benchmarks, outperforms models trained on 4.5x more data by 6.1%, and enables 1/3 data and half epochs to surpass baseline by 3.5%.", "output": "GRAPE significantly outperforms strong baselines across multiple models (Llama3.1-8B, Mistral-7B, Qwen2.5-7B). It achieves absolute gains up to 13.8% averaging across benchmarks compared to distilling from the strongest model, and maximum 17.3% improvements over training on 3 times more data. In realistic settings using post-training data for Tulu3 and Olmo-2, GRAPE outperforms baselines with 4.5 times the data by 6.1% and outperforms state-of-the-art data selection approaches by 3.9% on average. Remarkably, using only 1/3 of the data and half the number of epochs, GRAPE allows Llama3.1-8B to surpass Tulu3-SFT by 3.5%."},
+{"type": "english_summarize", "instruction": "Summarize the Dolma dataset and its role as a 3-trillion-token open corpus for LLM pretraining.", "input": "", "reasoning": "Dolma is a 3-trillion-token open corpus from Common Crawl, academic publications, code, and social media. It emphasizes transparency with detailed documentation of curation decisions.", "output": "Dolma is an open corpus of three trillion tokens for language model pretraining research, released by the Allen Institute for AI. It aggregates data from Common Crawl, academic publications (via S2ORC), code repositories (GitHub), and social media (Reddit, Wikipedia). The corpus emphasizes transparency, providing detailed documentation of all curation decisions including filtering criteria, deduplication methods, and tokenization strategies. This enables reproducible pretraining experiments and direct comparison of data curation techniques."},
+{"type": "english_summarize", "instruction": "Summarize the RedPajama-V2 dataset and its quality filtering approaches for open LLM pretraining.", "input": "", "reasoning": "RedPajama-V2 is an open dataset of 30 trillion tokens from Common Crawl, using multiple quality filters: exact deduplication, heuristics, classifier-based, and MinHash.", "output": "RedPajama-V2 is an open dataset for LLM pretraining, containing 30 trillion raw tokens from Common Crawl (July 2020–October 2023). It applies multiple quality filtering approaches: exact substring deduplication removing duplicated passages; heuristic quality scores based on document statistics; classifier-based filtering using fastText and a linear model; and MinHash deduplication with 5-grams. Processed subsets are available at different quality thresholds, enabling experiments on the trade-off between data quantity and quality. The dataset includes 38 billion quality-filtered tokens (high-quality subset) and 112 billion deduplicated tokens (medium-quality)."}