> For the complete documentation index, see [llms.txt](https://www.lumolabs.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://www.lumolabs.ai/lumo-dataset/about-lumo-iris.md).

# About Lumo-Iris

<figure><img src="/files/7WIourfH1ijvs3aYIRun" alt=""><figcaption></figcaption></figure>

The Lumo Iris DS Instruct dataset is a cornerstone for the Lumo large language model, designed to empower the model with a deep understanding of the Solana ecosystem. This next-generation dataset is 5x larger and more comprehensive than its predecessor, providing the foundation for Lumo's capabilities to answer questions, generate code, and assist users within the Solana domain.

**Knowledge cut-off date: 17th January, 2025**

{% hint style="info" %}
The dataset draws from an expanded and diverse range of authoritative sources within the Solana ecosystem, offering unparalleled depth and breadth of knowledge.
{% endhint %}

### Data Sources

The dataset integrates information from 15+ authoritative sources to ensure comprehensive coverage of the Solana ecosystem:

1. **Official Solana Documentation**\
   Comprehensive resources covering Solana's core concepts, protocols, and development tools.\
   Includes sections on.

* **Fundamentals:** Blockchain architecture, consensus mechanisms (Proof-of-History, Proof-of-Stake), tokenomics.
* **Development:** Smart contract development (using languages like Rust, Solidity), interacting with the Solana RPC, and using Solana developer tools.
* **Ecosystem:** DeFi protocols, NFTs, dApps, governance, and the broader Solana ecosystem.
* **Terminology:** Definitions of key terms and concepts within the Solana ecosystem.

2. **Project-Specific Documentation**

* **Jito:** Documentation for the Jito wallet and its associated features.
* **Raydium:** Documentation for the Raydium decentralized exchange (DEX) on Solana.
* **Jupiter:** Documentation for the Jupiter decentralized exchange aggregator.
* **Helius:** Documentation for the Helius Solana developer tools.
* **QuickNode:** Documentation for the QuickNode Solana infrastructure platform.
* **ChainStack:** Documentation for the ChainStack Solana infrastructure platform.
* **Meteora:** Documentation for the Meteora Solana infrastructure platform.
* **PumpPortal:** Documentation for the PumpPortal Solana-focused platform.
* **DexScreener:** Documentation for the DexScreener decentralized exchange explorer.
* **MagicEden:** Documentation for the MagicEden NFT marketplace.
* **Tatum:** Documentation for the Tatum blockchain APIs and tools.
* **Alchemy:** Documentation for Alchemy's blockchain infrastructure services.
* **Bitquery:** Documentation for Bitquery's blockchain data solutions.

<figure><img src="/files/zkLMuMbC6sT9DldOm1LN" alt=""><figcaption></figcaption></figure>

### Data Extraction and Processing

* **Data Extraction:**
  * Data was meticulously extracted from the designated sources using a combination of manual curation and automated techniques.
  * **Note:** The dataset was compiled with a strong emphasis on data integrity and accuracy. No automated scraping techniques were employed to avoid potential biases or inaccuracies.
* **Data Cleaning:**
  * **Removal of HTML/Markdown:** HTML tags, Markdown formatting, and other irrelevant formatting elements were removed to ensure clean and consistent text.
  * **Deduplication:** Duplicate entries were identified and removed to prevent redundancy and ensure data quality.
  * **Error Correction:** Minor spelling and grammatical errors were corrected to improve data consistency.
  * **Standardization:** Terminology was standardized across different sources to maintain consistency and improve data coherence.
* **Text Chunking:**
  * The extracted text was divided into smaller, manageable chunks of 1,500 characters with an overlap of 200 characters.
* **Question-Answer Pair Generation:**
  * For each chunk, 10 high-quality question-answer pairs were generated using a powerful language model (e.g., GPT-4).
  * The model was instructed to:
    * Generate questions that are relevant to the provided text chunk.
    * Ensure that the questions are answerable based solely on the information within the chunk.
    * Generate concise and informative answers that accurately reflect the content of the chunk.

<figure><img src="/files/8XF2mYNUIWqntGuClNnS" alt=""><figcaption></figcaption></figure>

### Dataset Structure

The Lumo Iris DS Instruct dataset is structured as a JSONL file, where each line represents a single question-answer pair. Each line contains the following fields:

* **`question`:** The question generated from the given text chunk.
* **`answer`:** The corresponding answer to the generated question.
* **`chunk`:** The original text chunk from which the question-answer pair was derived.