Christoforos Kapsalis

ITC6008 – Search Engines and Web Mining

Institution: MSc in Data Science, the American College of Greece

Term: Fall 2024

Full Codebase and Report: GitHub Repository

Overview

This repository houses our final submission “Lookup, Scrape, Deliver (LSD)”:
an end‑to‑end IR pipeline for the Azure Marketplace that combines

Project Structure

search_engines_submission/
├── front_end/                      # React.js user interface
│   ├── Dockerfile                  # builds the UI container
│   ├── package.json                # front‑end dependencies & scripts
│   ├── public/                     # static assets & HTML template
│   └── src/                        # React app code
│       ├── App.js                  # main component & routing
│       ├── VSRListing.js           # displays VSR results
│       └── ListingStaff.js         # displays RAG responses
│
├── rag_app/                        # Retrieval‑Augmented Generation API
│   ├── app.py                      # Flask server exposing /rag endpoint
│   ├── docker-compose.yml          # orchestrates RAG & ChromaDB services
│   ├── Dockerfile                  # builds the RAG container
│   └── requirements.rag.txt            # Python libs (flask, openai, chromadb, langchain)
│
├── vsr_app/                        # Vector Space Retrieval API (un‑dockerized)
│   ├── crawler/                    # optional stand‑alone crawler
│   │   ├── 1query_creation.py      # seed‑URL builder from filter maps
│   │   └── 2scraper.py             # Playwright scraper for app details
│   ├── data/                       # input mappings & temporary outputs
│   │   ├── input/                  # filter & category CSVs
│   │   └── output/                 # final query→result mapping CSV
│   ├── vector_space_retrieval_csv/ # core VSR engine
│   │   ├── vsr/                    # Python package
│   │   │   ├── __main__.py         # “python -m vsr” entrypoint
│   │   │   ├── common/             # tokenization, cleaning, helpers
│   │   │   ├── global.cfg          # paths, thresholds, flags
│   │   │   └── modules/            # indexer, inverted_index, metrics, search, query_processor
│   │   ├── tests/                  # integration tests (uses sample CSVs)
│   ├── README.md                   # VSR‑specific overview
│   └── requirements.main.txt            # top‑level dependencies (flask, vsr engine)
│
├── vsr_dockerized_response/        # Dockerized VSR deployment
│   ├── app.py                      # Flask server wrapping vsr_app core
│   ├── Dockerfile                  # builds VSR container for production
│   └── FlaskVSR_instruction.docx   # how to run & test the container
│   └── requirements.vsr.txt            # Python libs (flask, openai, chromadb, langchain)
|
├── ITC6008A1-Christodoulidis_…_Kounelis.docx  
│                                   # consolidated project report :contentReference[oaicite:3]{index=3}
├── Final_Presentation_ITC6008_Search_Engines.pptx  
│                                   # slide deck summarizing architecture & results
└── README.md                       # ← you are here

Component Descriptions

  1. front_end/ (React UI)
    • App.js: mounts the query form, toggles VSR vs RAG mode, requests backend.
    • VSRListing.js: renders the top‑N VSR results (title, snippet, link, score).
    • ListingStaff.js: displays the chatbot‑style RAG response panel.
    • Dockerfile: containerizes the React build for easy deployment.
  2. rag_app/ (RAG backend)
    • app.py: Flask REST API (POST /rag)
      • loads ChromaDB vector store
      • on request: retrieves top‑k chunks, invokes OpenAI GPT‑3.5‑turbo
      • returns a context‑aware answer
    • docker-compose.yml: brings up ChromaDB + RAG service
    • Dockerfile: builds Python image with all RAG dependencies
    • requirements.rag.txt: flask, openai, langchain, chromadb, etc.
  3. vsr_app/ (VSR backend)
    • crawler/
      • 1query_creation.py: reads azure_marketplace_*_mapping.csv, composes seed URLs.
      • 2scraper.py: uses Playwright to scrape app titles, prices, descriptions.
    • data/input/
      • azure_marketplace_catalog_mapping.csv: category → query mapping
      • azure_marketplace_filter_mapping.csv: filter options → URL params
    • data/output/query_mapping.csv: final CSV linking seed URLs to scraped app data
    • vector_space_retrieval_csv/vsr/
      • __main__.py: CLI entry (index & serve).
      • common/: cleaning, tokenization, DTM helpers.
      • global.cfg: path configs, TOP_K setting.
      • modules/:
        • indexer/: builds TF‑IDF / inverted index.
        • inverted_index/: serializes index to disk.
        • query_processor/: query → vector conversion.
        • search/: cosine similarity, ranking logic.
        • metrics/: computes MAP, Precision@K for evaluation.
      • vsr_app/requirements.txt: flask, pandas, scikit-learn, playwright, etc.
      • tests/integration_test.py: smoke tests for VSR endpoint.
  4. vsr_dockerized_response/ (Production VSR)
    • app.py: wraps vsr_app core in a single Flask service
    • Dockerfile: multi‑stage build:
    • installs vsr_app/ dependencies
    • copies code & runs app.py
    • FlaskVSR_instruction.docx: step‑by‑step to build & run container

How to Run

  1. Extract & rename your four archives under one root 
    mv front_end/ rag_app/ vsr_app/ VSR_Dockerized_Response/ ./search_engines_submission/
  2. RAG 
    cd rag_app
docker-compose up --build
  3. VSR (dockerized) 
    cd VSR_Dockerized_Response
docker build -t vsr-service .
docker run -p 5000:5000 vsr-service
  4. Frontend 
    cd front_end
docker build -t ir-ui .
docker run -p 3000:3000 ir-ui
  5. Or run un‑dockerized VSR for development 
    cd vsr_app
pip install -r requirements.txt
python crawler/2scraper.py         # (optional) refresh data
python -m vector_space_retrieval_csv.vsr
flask run                         # if you’ve wired app.py

Reports & Slides

Notes

← Back to projects