MetaCommit

data/LLM_English/instruction_following.jsonl

@@ -0,0 +1,19 @@
+{"type": "english_instruction", "instruction": "Write a short paragraph describing the required paradigm shift in data curation as outlined in the introduction. Use formal academic tone.", "input": "", "reasoning": "The shift is from monolithic web scraping to a multi-stage pipeline: foundational pre-training for syntax, instruction tuning for reasoning, stylistic alignment for prose, and pragmatic disambiguation for sarcasm/idioms.", "output": "The necessary paradigm shift entails moving beyond monolithic web scraping toward a meticulously stratified data pipeline. This architecture comprises progressive stages: foundational pre-training for structural syntax, targeted instruction tuning for academic rigor and deep reasoning, stylistic alignment for world-class literary prose, and pragmatic disambiguation to handle sarcasm, humor, idioms, and profanity. Only through such a curated curriculum can an LLM transcend stochastic parroting and emerge as a nuanced, deeply thoughtful linguistic entity."},
+{"type": "english_instruction", "instruction": "Write a technical note describing the curation pipeline of FineWeb-Edu, including the model used for annotation and the scoring mechanism.", "input": "", "reasoning": "Pipeline: Common Crawl → FineWeb (15T tokens) → sample 460k texts → annotate with Llama-3-70B-Instruct (additive score 0-5) → train linear regressor on Snowflake embeddings → predict scores for all → filter score ≥3 → FineWeb-Edu (5.4T tokens).", "output": "Curation pipeline for FineWeb-Edu: 1) Start from FineWeb, a 15-trillion-token corpus derived from 96 Common Crawl snapshots. 2) Sample 460,000 candidate texts. 3) Use Llama-3-70B-Instruct to annotate each text with an additive educational score (0-5). 4) Train a linear regressor on Snowflake-arctic-embed-m embeddings to predict scores for the entire FineWeb corpus. 5) Apply a threshold of 3 or higher. 6) Retain 5.4 trillion tokens as FineWeb-Edu. This automated, scalable filtering prioritizes academic and structured content over noisy web text."},
+{"type": "english_instruction", "instruction": "Write a brief summary of the structural diversity datasets mentioned after Common Corpus (The Pile, GneissWeb, ProX framework). Include their sizes and key features.", "input": "", "reasoning": "The Pile: 825 GB, 22 domains (arXiv, PubMed, GitHub). GneissWeb: ~10T tokens, exact substring deduplication + quality filters. ProX: 0.3B-parameter LM that writes cleaning programs per record.", "output": "The Pile (825 GB) combines 22 diverse high-quality sources including academic publications (arXiv, PubMed Central), GitHub repositories, and medical text, improving cross-disciplinary vocabulary. GneissWeb (~10 trillion tokens) uses exact substring deduplication and an ensemble of quality filters, outperforming FineWeb-V1.1.0 by 2.73 percentage points across 11 benchmarks. The ProX framework (Programming Every Example) uses a lightweight 0.3B-parameter language model to write and execute targeted cleaning programs (e.g., HTML stripping, string normalization) for each record in datasets like C4, RedPajama-V2, and DCLM, ensuring a pristine foundational corpus."},
+{"type": "english_instruction", "instruction": "Write a technical description of the C4_200M synthetic dataset and its purpose in grammatical error correction training.", "input": "", "reasoning": "C4_200M contains 200 million sentence pairs with synthetically induced grammatical errors. It is massive and complements real learner corpora by providing controlled error types at scale.", "output": "The C4_200M synthetic dataset comprises 200 million sentence pairs where grammatical errors have been artificially introduced into clean text. Each pair consists of an error-containing sentence and its corrected version. The synthetic generation allows controlled sampling of specific error types (e.g., tense mismatches, subject-verb disagreement, punctuation errors). This dataset serves as a massive complement to real learner corpora like Lang-8, enabling the LLM to encounter a diverse, balanced distribution of grammatical mistakes. Because native and non-native error patterns differ, C4_200M ensures the model is robust against all forms of input degradation, making it capable of correcting errors from any user population."},
+{"type": "english_instruction", "instruction": "Write a technical explanation of the SCAR framework and how it selects training data for stylistic consistency.", "input": "", "reasoning": "SCAR (Style Consistency-Aware Response Ranking) ranks instruction-response pairs by linguistic form and instructional surprisal, selecting the most style-consistent examples. Sometimes only 0.7% of data is needed to match full-dataset performance.", "output": "The SCAR (Style Consistency-Aware Response Ranking) framework automates data selection for stylistic tuning. It operates on two key elements: (1) linguistic form — the surface features of text (punctuation, sentence length, register markers); and (2) instructional surprisal — how unexpected the response is given the instruction. SCAR ranks all instruction-response pairs in a training set by their stylistic consistency, then selects the most consistent examples. In practice, fine-tuning LLMs on as little as 0.7% of the full dataset — the highest-ranked examples — matches or surpasses the performance of models trained on entire, uncurated datasets. This efficiency comes from eliminating stylistically inconsistent or noisy examples that would otherwise degrade the model's register adherence."},
+{"type": "english_instruction", "instruction": "Write a technical description of the FACT and RACE evaluation frameworks introduced in DeepResearch Bench.", "input": "", "reasoning": "DeepResearch Bench provides 100 PhD-level research tasks across 22 fields. RACE (Reference-based Adaptive Criteria-driven Evaluation) assesses report quality. FACT (Framework for Factual Abundance and Citation Trustworthiness) measures citation count and accuracy.", "output": "DeepResearch Bench introduces two novel evaluation frameworks for assessing deep research agents. RACE (Reference-based Adaptive Criteria-driven Evaluation) evaluates the quality of generated research reports by adaptively applying criteria based on reference answers — it measures coherence, completeness, and methodological soundness. FACT (Framework for Factual Abundance and Citation Trustworthiness) assesses the model's citation practices through two metrics: effective citation count (how many claims are properly supported) and overall citation accuracy (whether citations actually support the claims they accompany). By fine-tuning LLMs on data scoring highly on FACT, the model learns to automatically embed context, origin, and mechanism into its answers, mimicking the rigorous citation standards of peer-reviewed literature rather than generating unsupported assertions."},
+{"type": "english_instruction", "instruction": "Write a technical description of the LongPage dataset and how it enables hierarchical narrative planning over extended contexts.", "input": "", "reasoning": "LongPage provides 300 full-length books (40k-600k tokens each) with hierarchical reasoning traces: character archetypes, world rules, scene breakdowns, thematic coherence. These act as 'Chain of Thought for creative writing' over massive context windows.", "output": "The LongPage dataset addresses long-horizon reasoning for creative writing by providing 300 full-length books, each ranging from 40,000 to over 600,000 tokens. Crucially, each book is accompanied by hierarchical reasoning traces — structured annotations that map narrative logic across the entire text. These traces include character archetypes (personality consistency), world rules (setting constraints), scene breakdowns (event sequencing), and thematic coherence (motif tracking). Functionally, these traces act as 'Chain of Thought for creative writing,' explicitly showing the model how to plan and maintain logical consistency over massive context windows. Training on LongPage ensures the model does not lose its train of thought when deep-diving into complex, multi-page English responses — whether writing a novel, a legal brief, or a philosophical treatise."},
+{"type": "english_instruction", "instruction": "Write a technical description of the Wiki Neutrality Corpus (WNC) and its role in teaching objective, academic prose.", "input": "", "reasoning": "WNC is used for neutralizing subjectivity bias. It teaches the model to identify emotionally loaded terms (e.g., 'claimed' vs 'stated', 'controversial' vs 'notable') and replace them with objective, academic equivalents.", "output": "The Wiki Neutrality Corpus (WNC) is a Text Style Transfer dataset specifically designed for neutralizing subjectivity bias in writing. It consists of pairs of Wikipedia sentences: one version containing biased or emotionally loaded language, and a neutralized version that adheres to Wikipedia's Neutral Point of View policy. Training on WNC teaches the LLM to identify subjective language patterns — loaded adjectives ('horrific', 'brilliant'), framing verbs ('claimed' vs 'stated', 'admitted' vs 'explained'), and evaluative hedges ('controversially', 'fortunately'). The model learns to replace these with objective, academic equivalents: 'claimed' becomes 'stated', 'horrific event' becomes 'event', 'fortunately' is removed entirely. This capability is essential for generating scholarly prose that prioritizes factual reporting over rhetorical persuasion, a hallmark of world-class academic English."},
+{"type": "english_instruction", "instruction": "Write a technical description of the Ironic Corpus and how it distinguishes sarcasm from situational irony.", "input": "", "reasoning": "The Ironic Corpus provides human-annotated labels distinguishing situational irony (events contradicting expectations) from sarcasm (verbal statements contradicting intended meaning). This prevents models from conflating all non-literal text with sarcasm.", "output": "The Ironic Corpus is a human-annotated dataset designed to distinguish between closely related but distinct rhetorical devices. It provides fine-grained labels separating situational irony (an event or outcome that contradicts expected or intended results — e.g., a fire station burning down) from verbal sarcasm (a statement whose literal meaning opposes the speaker's intended meaning — e.g., 'Great job' after a failure). This distinction is crucial because both phenomena involve contradiction, but they operate at different levels: irony at the level of events, sarcasm at the level of utterance meaning. Without explicit training on this distinction, LLMs conflate all non-literal, contradictory-sounding text with sarcasm, leading to false positives. The Ironic Corpus provides the annotation granularity needed for the model to correctly classify each device based on whether the contradiction is situational or verbal, enabling precise pragmatic understanding across diverse rhetorical forms."},
+{"type": "english_instruction", "instruction": "Write a technical description of the SlangLLM framework and the OpenSubtitles-Slang dataset for training contemporary colloquial understanding.", "input": "", "reasoning": "OpenSubtitles-Slang provides slang in messy real-world conversational contexts from movie/TV subtitles. SlangLLM is a framework for dynamic detection and contextual filtering of slang, mapping rapidly evolving internet vernacular.", "output": "The SlangLLM framework, trained on the OpenSubtitles-Slang dataset, addresses the challenge of rapidly evolving contemporary slang in natural language processing. OpenSubtitles-Slang extracts slang usage from messy, real-world conversational contexts within movie and television subtitles — capturing how slang actually appears in natural dialogue (with interruptions, false starts, overlapping speech, and register mixing). This contrasts with curated or dictionary-based slang datasets that present sanitized, context-free definitions. SlangLLM then implements dynamic detection (identifying which words or phrases function as slang in a given context) and contextual filtering (determining when slang is appropriate to generate vs. when formal language is required). This training ensures the model understands the fluidity of modern colloquialisms — including internet-born terms, regional expressions, and generational slang — preventing it from sounding overly robotic, rigid, or anachronistic when users prompt it to adopt a casual, friendly persona."},
+{"type": "english_instruction", "instruction": "Write a technical description of the Pragmatic Category Table from the document, summarizing all six pragmatic dimensions and their datasets.", "input": "", "reasoning": "Table covers Sarcasm/Irony (SARC, PodSarc), Humor/Satire (Deceptive Humor Dataset), Idioms/Slang (MAGPIE, SlangLLM), Profanity/Toxicity (Surge AI, Jigsaw), each with specific annotation structures and proficiency enhancements.", "output": "The Pragmatic Category Table from the document organizes six dimensions of non-literal language mastery for LLMs. (1) Sarcasm/Irony: SARC (1.3M Reddit comments) and PodSarc (9.6-second podcast segments) teach detection of opposite intended meaning via extended conversational context. (2) Humor/Satire: Deceptive Humor Dataset provides Satire Levels 1-3 across five archetypes (Dark Humor, Irony, Social Commentary, Wordplay, Absurdity), enabling generation and comprehension of subtle wordplay, dark humor, and social commentary. (3) Idioms: MAGPIE provides 56,000+ instances of 1,756 unique idioms across varied contexts, preventing literal misinterpretation of semantic opacity. (4) Slang: SlangLLM with OpenSubtitles-Slang maps contemporary internet colloquialisms, ensuring fluid casual interaction. (5) Profanity: Surge AI catalogs 1,600+ profanities with severity scale 1-3 and semantic target categorization. (6) Toxicity: Jigsaw provides 160,000 labeled comments with sub-labels (severe toxicity, insults, threats, identity-based hate). Together, these six dimensions enable an LLM to understand language as it is actually used — with all its irony, humor, idiom, slang, and emotional intensity — while maintaining ethical boundaries."},
+{"type": "english_instruction", "instruction": "Write a final synthesis paragraph describing the ultimate vision of an LLM trained according to this document's paradigm.", "input": "", "reasoning": "The vision is a model that transcends 'stochastic parrot' limitations to become an 'elite, deeply thoughtful, highly nuanced linguistic entity' — capable of graduate-level reasoning, flawless grammar, philosophical contemplation, native-like pragmatics, and ethical discernment across all registers of English.", "output": "The ultimate vision of an LLM trained according to this document's paradigm is a model that transcends the limitations of a stochastic parrot — a system that does not merely predict plausible next tokens but genuinely understands English at a PhD level. This model would demonstrate: graduate-level reasoning on GPQA and MMLU-Pro without relying on shallow heuristics; flawless grammatical precision including mastery of bridging anaphora and complex multi-clause dependencies; deep philosophical contemplation through Socratic dialogue and ethical reasoning; native-like pragmatic competence including accurate sarcasm detection, context-appropriate humor generation, idiom comprehension, and fluid slang usage; and ethical discernment that distinguishes casual banter from targeted harassment. It would shift seamlessly from the ornate elegance of Victorian prose to the minimalist clarity of technical documentation, from the formal register of legal argumentation to the casual warmth of friendly conversation. In short, it would emerge not as a brittle text generator but as an elite, deeply thoughtful, highly nuanced linguistic entity — a model that truly understands English better than the average human and communicates with world-class proficiency, wisdom, and wit."},
+{"type": "english_instruction", "instruction": "Write a technical note describing the classifier training details for FineWeb-Edu, including dataset size, epochs, and F1 scores.", "input": "", "reasoning": "Classifier trained on 450k Llama3 annotations, 20 epochs, learning rate 3e-4, embedding layers frozen. Achieves 82% binary F1 at threshold 3.", "output": "The FineWeb-Edu classifier is trained on 450,000 samples annotated by Llama-3-70B-Instruct with educational scores 0–5. A classification head with single regression output is added to Snowflake-arctic-embed-m. Training runs for 20 epochs with learning rate 3e-4 while embedding and encoder layers remain frozen. The model achieves 82% F1 score when converted to a binary classifier using threshold 3. Weighted average F1 across all score levels is 0.71, with individual class performance varying due to annotation noise."},
+{"type": "english_instruction", "instruction": "Write a technical description of the language coverage in Common Corpus beyond English.", "input": "", "reasoning": "Common Corpus includes major European languages plus low-resource languages (Arabic, Bengali, Latin, Persian, Russian, Sanskrit, Urdu). Open Science is 85% English; Open Culture spans 13+ languages.", "output": "Common Corpus provides multilingual coverage ranging from high-resource European languages to low-resource languages rarely represented in pretraining datasets. Open Culture spans at least 13 major languages (French, English, German, Spanish, Portuguese, Italian, Dutch, Luxembourgish, Danish, Swedish, Serbian, Czech, Greek) plus significant representation in Arabic, Bengali, Latin, Persian, Russian, Sanskrit, and Urdu. Open Science focuses primarily on English (approximately 85%) with substantial French, Spanish, and German content. Open Semantic provides Wikidata natural language renditions in 300 languages. This breadth enables cross-lingual transfer learning and supports the model's understanding of English through comparative linguistic structures."},
+{"type": "english_instruction", "instruction": "Write a technical note on the inline annotation features of S2ORC full text (citations, figures, tables).", "input": "", "reasoning": "Full text annotated with automatically-detected inline mentions of citations, figures, and tables, each linked to corresponding paper objects.", "output": "S2ORC's structured full text includes automatically-detected inline annotations for three key elements. Citation mentions: any reference to another paper (e.g., '(Smith, 2020)') is detected and linked to the resolved bibliographic entry in S2AG. Figure mentions: references to figures within the paper (e.g., 'as shown in Figure 3') are annotated and linked to the figure's location. Table mentions: references to data tables are similarly annotated and linked. These annotations enable the model to learn the discourse functions of citations (supporting claims), figures (visual evidence), and tables (data presentation), critical for generating authentic scientific writing."},
+{"type": "english_instruction", "instruction": "Write a technical description of how bridging anaphora is encoded in CoNLL-U's MISC field using CorefUD standards.", "input": "", "reasoning": "Bridging-specific information is captured by attribute-value pairs in the MISC column, following CorefUD 1.3 conventions for linking anaphors to their implicit antecedents.", "output": "In the CorefUD (Coreference in Universal Dependencies) framework, bridging anaphora annotations are stored as attribute-value pairs in the MISC column of CoNLL-U files. For a bridging anaphor like 'the door' in 'There is a house. The door is red', the annotation includes a bridging link to the previously mentioned 'house' using attributes such as 'Bridge=house' or 'Referent=house'. This encoding allows explicit representation of the inferential relationship between entities that are not coreferential (the door is not the same as the house) but are semantically linked. Multiple bridging relations can be encoded per token, enabling training of LLMs to resolve implicit references across sentence boundaries."},
+{"type": "english_instruction", "instruction": "Write a technical description of GRAPE's response selection process for a single instruction, contrasting it with standard SFT data collection.", "input": "", "reasoning": "Standard SFT collects responses from external LLMs without distribution matching. GRAPE computes target model's perplexity for each candidate response and selects the lowest-perplexity (highest-probability) one.", "output": "In standard SFT data collection, an instruction is paired with a response sampled from a strong external LLM (e.g., GPT-4) or human annotators, regardless of whether that response matches the target model's distribution. In GRAPE, for each instruction, the system first gathers multiple candidate responses from various sources (multiple LLMs, humans, or both). For each candidate, the target model computes the log probability (or perplexity) of the response given the instruction. The candidate with the highest probability (lowest perplexity) is selected — this is the response the target model 'most wants to generate' based on its pretrained distribution. The selected instruction-response pair is then added to the SFT dataset. This ensures every training example is distributionally congruent with the model, minimizing the need for weight adjustment during fine-tuning."},
+{"type": "english_instruction", "instruction": "Write a technical note on the tokenization and deduplication strategies used in Dolma, based on 2025 documentation.", "input": "", "reasoning": "Dolma uses MinHash deduplication with 13-gram shingles and 16 hash functions per band. Tokenization uses GPT-NeoX-20B tokenizer with 50,257 vocabulary size.", "output": "Dolma implements per-document deduplication using MinHash with 13-gram shingles, 16 hash functions per band, and Jaccard similarity threshold of 0.8. This removes near-duplicate documents while preserving legitimate similar content. For tokenization, Dolma uses the GPT-NeoX-20B tokenizer with a vocabulary size of 50,257 tokens, applying byte-level byte-pair encoding (BPE). The corpus also includes URL-based filtering to remove adult content, language identification using fastText with confidence threshold 0.8 (English only), and heuristic filters removing documents with excessive repetition or outlier lengths."},
+{"type": "english_instruction", "instruction": "Write a technical description of DCLM (Data Compaction for Language Models) and its integration with ProX for scalable refinement.", "input": "", "reasoning": "DCLM is a framework for compacting training data by removing redundancy. ProX provides sample-wise cleaning before compaction, ensuring DCLM operates on pristine text.", "output": "DCLM (Data Compaction for Language Models) is a framework that reduces pretraining data size while preserving informational content. It identifies and removes redundant passages within and across documents using semantic similarity hashing and perplexity-based importance scoring. DCLM can reduce dataset size by 50-80% while maintaining or improving downstream task performance. ProX integrates with DCLM as a preprocessing step: ProX first performs sample-wise cleaning (HTML stripping, normalization, anomaly removal) on raw documents. The cleaned output is then fed to DCLM for compaction. This two-stage pipeline ensures that DCLM operates on clean, standardized text, preventing artifacts from being mistakenly identified as informative content and retained."}