Have build a price comparison website that displays products and their prices from several different websites. I have attached files giving every single detail, The FILE includes what languages to...

1 answer below »
Have build a price comparison website that displays products and their prices from several different websites. I have attached files giving every single detail, The FILE includes what languages to use, what website to use for web scrapping, how the website should look, what functionality the website should have, what database to use and its structure. The file also includes what files I should receive at the end when its done. Also one last thing please complete this project with the requested requirements as last time I did not receive what I wanted and failed the project. Please read the file properly and see what I want before the work is being started on, as it has to be done with the requirements I have given and for example cannot change the language like last time at the very last minute.


Number of Pages 4 Have to build a price comparison website that displays products and their prices from several different websites. The products that need to be compared are mobile phones from different websites. • The user can search for products on the website. • Shopping functionality and customer registration/login are not required. • Your website provides links that enable users to navigate to the product on the original website. • Website scraping must be done using Java. • You must use a SQL database to store your data. MySQL should be used. • Several threads should be used to download data. • You must use Spring, Hibernate and Maven. • The back end of the website must be implemented in JavaScript running on Node.js. Cannot use any other programming languages. • Your application includes a REST API. This can be integrated into your front end (via AJAX). Or it can be a separate web service that enables third parties to access your data. • The front end must be written in HTML, CSS and JavaScript. You cannot write an app. • There are marks available for the quality and quantity of data that you have scraped. • Marks are available for code quality, unit testing and the front end of the website. Number of Pages 4 This is an EXAMPLE of what the website could/should look like: This will be the Homepage of the Website. It will include a search bar that will allow user to search for a specific product. Number of Pages 4 This is the product page. After a search for a product, they will be directed to this page, which will include the product details, list of websites/store selling the same product and their prices Database Design: This is the design of the database, of which the data from the websites will be stored. It will be a MYSQL database and will have 3 different tables to make data more organized and accessible. Number of Pages 4 When you upload the project back to ME it SHOULD include the following: Source code: • Maven build file (pom.xml). • Spring configuration file(s) (beans.xml). Not needed if you are using annotations. • Hibernate configuration file(s). • Java source code for web scraping. • Java source code for unit tests. • Java API documentation. • Node.js JavaScript source code. • JavaScript source code for unit tests. • Front end source code for website (JavaScript, HTML, CSS). • Other source files for website, for example. Vue files etc. Database dump: • Back up your database using mysqldump Websites to use to scrap data from: Website URLs Amazon https://www.amazon.com EE https://ee.co.uk John Lewis https://www.johnlewis.com AO https://www.ao-mobile.com Fonehouse https://www.fonehouse.co.uk Number of PAGES 2 How the project will be marked and the POINTS it MUST meet and shown below: Number of PAGES 2
Answered 8 days AfterApr 16, 2021

Answer To: Have build a price comparison website that displays products and their prices from several different...

Swapnil answered on Apr 24 2021
140 Votes
Price Comparison Website/Back-end/scraper/pom.xml
Price Comparison Website/Back-end/scraper/src/main/java/AppConfig.java
Price Comparison Website/Back-end/scraper/src/main/java/AppConfig.java
import java.util.ArrayList;
import java.util.List;
import org.hibernate.SessionFactory;
import org.hibernate.boot.MetadataSources;
import org.hibernate.boot.registry.StandardServiceRegistry;
import org.hibernate.boot.registry.StandardServiceRegistryBuilder;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
@Configuration
public class AppConfig {
    SessionFactory sessionFactory;

    //Create a bean instance of the ScraperManager class to run each one
    @Bean
    public ScraperManager scraperManager(){
        ScraperManager scraperManager = new ScraperManager();

        //Create list of web scrapers and add to scraper manager
        List scraperList = new ArrayList();
        scraperList.add(cargiant());
        scraperList.add(carhub());
        scraperList.add(classicCars());
        scraperList.add(hrowen());
        scraperList.add(lamborghini());
        scraperList.add(rybrook());
        scraperManager.setScraperList(scraperList);
        return scraperManager;
    }

    //Create a bean instance of the Cargiant class
    @Bean
    public Cargiant cargiant(){
        Cargiant cargiant = new Cargiant();
        cargiant.setCarDao(carDao());
        cargiant.setCrawlDelay(1);
      return cargiant;
    }

    //Create a bean instance of the Carhub class
    @Bean
    public Carhub carhub(){
        Carhub carhub = new Carhub();
        carhub.setCarDao(carDao());
        carhub.setCrawlDelay(1);
        return carhub;
    }

    //Create a bean instance of the Classiccars class
    @Bean
    public Classiccars classicCars(){
        Classiccars classicCars = new Classiccars();
        classicCars.setCarDao(carDao());
        classicCars.setCrawlDelay(1);
        return classicCars;
    }

    //Create a bean instance of the Hrowen class
    @Bean
    public Hrowen hrowen(){
        Hrowen hrowen = new Hrowen();
        hrowen.setCarDao(carDao());
        hrowen.setCrawlDelay(1);
        return hrowen;
    }

    //Create a bean instance of the Lamborghini class
    @Bean
    public Lamborghini lamborghini(){
        Lamborghini lamborghini = new Lamborghini();
        lamborghini.setCarDao(carDao());
        lamborghini.setCrawlDelay(1);
        return lamborghini;
    }

    //Create a bean instance of the Lamborghini class
    @Bean
    public Rybrook rybrook(){
        Rybrook rybrook = new Rybrook();
        rybrook.setCarDao(carDao());
        rybrook.setCrawlDelay(1);
        return rybrook;
    }

    //Create a bean instance of the CarDao class
    @Bean
    public CarDao carDao(){
        CarDao carDao = new CarDao();
        carDao.setSessionFactory(sessionFactory());
        return carDao;
    }

    //Create a session factory bean
    @Bean
    public SessionFactory sessionFactory(){
        if(sessionFactory == null){//Build sessionFatory once only
            try {
                //Create a builder for the standard service registry
                StandardServiceRegistryBuilder standardServiceRegistryBuilder = new StandardServiceRegistryBuilder();
                //Load configuration from hibernate configuration file.
                //Here we are using a configuration file that specifies Java annotations.
                standardServiceRegistryBuilder.configure("resources/hibernate-annotations.cfg.xml"); 
                //Create the registry that will be used to build the session factory
                StandardServiceRegistry registry = standardServiceRegistryBuilder.build();
                try {
                    //Create the session factory - this is the goal of the init method.
                    sessionFactory = new MetadataSources( registry ).buildMetadata().buildSessionFactory();
                }
                catch (Exception e) {
                        /* The registry would be destroyed by the SessionFactory, 
                            but we had trouble building the SessionFactory, so destroy it manually */
                        System.err.println("Session Factory build failed.");
                        e.printStackTrace();
                        StandardServiceRegistryBuilder.destroy( registry );
                }
                //Ouput result
                System.out.println("Session factory built.");
            }
            catch (Throwable ex) {
                // Make sure you log the exception, as it might be swallowed
                System.err.println("SessionFactory creation failed." + ex);
                ex.printStackTrace();
            }
        }
        return sessionFactory;
    }
}
Price Comparison Website/Back-end/scraper/src/main/java/CarDao.java
Price Comparison Website/Back-end/scraper/src/main/java/CarDao.java
import org.hibernate.Session;
import org.hibernate.SessionFactory;
public class CarDao {
    SessionFactory sessionFactory;

    /** Adds a new car to the database */
    public void saveCar(CarXML car){

        //Get a new Session instance from the SessionFactory
        Session session = sessionFactory.getCurrentSession();
        //Start transaction
        session.beginTransaction();
        //Add Car to database - will not be stored until we commit the transaction
        session.save(car);
        //Commit transaction to save it to database
        session.getTransaction().commit();

        //Close the session and release database connection
        session.close();
        System.out.println("Car added to database with ID: " + car.getId());
    }

        /** Adds a new car to the database */
    public void addUrl(Url url){

        //Get a new Session instance from the SessionFactory
        Session session = sessionFactory.getCurrentSession();
        //Start transaction
        session.beginTransaction();
        //Add Car to database - will not be stored until we commit the transaction
        session.save(url);
        //Commit transaction to save it to database
        session.getTransaction().commit();

        //Close the session and release database connection
        session.close();
        System.out.println("Car added to database with ID: " + url.getId());
    }
    //Getter and Setter for the session factory
    public SessionFactory getSessionFactory() {
        return sessionFactory;
    }
    public void setSessionFactory(SessionFactory sessionFactory) {
        this.sessionFactory = sessionFactory;
    }
}
Price Comparison Website/Back-end/scraper/src/main/java/CarDaoTest.java
Price Comparison Website/Back-end/scraper/src/main/java/CarDaoTest.java
import java.util.ArrayList;
public class CarDaoTest extends CarDao {
    ArrayList carArrayList = new ArrayList();
    ArrayList urlArrayList = new ArrayList();

    public void saveCar(CarXML cx){
        carArrayList.add(cx);
    } 

    public void addUrl(Url url){
        urlArrayList.add(url);
    }

}
Price Comparison Website/Back-end/scraper/src/main/java/Cargiant.java
Price Comparison Website/Back-end/scraper/src/main/java/Cargiant.java
import java.io.IOException;
import java.net.MalformedURLException;
import java.net.URL;
import java.util.logging.Level;
import java.util.logging.Logger;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
public class Cargiant extends WebScraper {

    //Url for the website I will be scraping
    String webUrl;

    @Override
    public void run(){
        runThread = true;

        while(runThread){

            //Scrape data from pages 1-5
            for (int i = 1; i < 6; i++) {
 
                //webUrl add the incrementation of pages to the link
                webUrl = "https://www.cargiant.co.uk/search/all/all/page/" +  i;
                System.out.println("This is the foor looop; " + webUrl);
                System.out.println("Cargiant thread is scraping data.");
 
                //Document 1 to scrape data from
                Document document1 = null;
                  try {
                       document1 = Jsoup.connect(webUrl).timeout(6000).get();
                  } catch (IOException ex) {
                      Logger.getLogger(Cargiant.class.getName()).log(Level.SEVERE, null, ex);
                    }
 
                //Grab the div that wraps the data
                Elements ele = document1.select("section.results");
                for (Element row : ele.select(".vehicle")) {
                    //Grap elements you want to scrape from each row
                    String carUrl = row.select(".vehicle-cover").attr("href");
                    String imageUrl = row.select(".image").attr("style");
                    String description = row.select(".small-7").text();
                    String features = row.select(".car-features").text();
                    String price = row.select(".car-price").text();
                    //Split the URL and add it to the database table
                    try {
                        URL urlSplit = new URL("https://www.cargiant.co.uk".concat(carUrl));
 
                        //Create a Url class instance
                        Url url = new Url();
                        url.setDomain(urlSplit.getHost());
                        url.setPath(urlSplit.getPath());
                        url.setQuery_String("");
 
                        //Save Url in carDao
                        carDao.addUrl(url);
 
                        //Create an istance of the CarXML
                        CarXML car = new CarXML();

                        //Set values of Car class that we want to add
                        car.setImage_url(imageUrl);
                        car.setDescription(description);
                        car.setFeatures(features);
                        car.setPrice(price);
                        car.setUrl(url);

                        //Save car in carDao
                        carDao.saveCar(car);

                    } catch (MalformedURLException ex) {
                        Logger.getLogger(Cargiant.class.getName()).log(Level.SEVERE, null, ex);
                    }
                    //Print results to console
                    System.out.println("Car Url: " + carUrl + "\n" +
                            "Image_url: " + imageUrl + "\n" +
                            "Description: " + description + "\n" +
                            "Features: " + features + "\n" +
                            "Price: " + price + "\n" 
                    );
                } 
            }
            //Sleep for the crawl delay, which is in seconds
            try{
                sleep(1000 * crawlDelay);//Sleep is in milliseconds, so we need to multiply the crawl delay by 1000
            }
            catch(InterruptedException ex){
                System.err.println(ex.getMessage());
            }
            break;
        }
    }
}
Price Comparison Website/Back-end/scraper/src/main/java/Carhub.java
Price Comparison Website/Back-end/scraper/src/main/java/Carhub.java
import java.io.IOException;
import static java.lang.Thread.sleep;
import java.net.MalformedURLException;
import java.net.URL;
import java.util.logging.Level;
import java.util.logging.Logger;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
public class Carhub extends WebScraper {

    @Override
    public void run(){
        runThread = true;

        //While loop will keep running until runThread is set to false;
        while(runThread){
            System.out.println("Carhub thread is scraping data.");
 
            Document document2 = null;
            try {
                   document2 = Jsoup.connect("https://www.carhublondon.co.uk/used-cars").timeout(6000).get();
            } catch (IOException ex) {
                  Logger.getLogger(Carhub.class.getName()).log(Level.SEVERE, null, ex);
                } 
 
            //Grab the div that wraps the data
            Elements ele = document2.select("section.car-search-results");
            for (Element row : ele.select(".end")) {
 
                //Grap elements you want to scrape from each row
                String carUrl = row.select(".car-search-excerpt a").attr("href");
                String imageUrl = row.select(".car-picture-title-wrap").attr("style");
                String description = row.select(".title-row").text();
                String price = row.select(".price-row").text();
 
                //Split the URL and add it to the database table
                try {
                    URL urlSplit = new URL("https://www.carhublondon.co.uk".concat(carUrl));
 
                    //Create a Url class instance
                    Url url = new Url();
                    url.setDomain(urlSplit.getHost());
                    url.setPath(urlSplit.getPath());
                    url.setQuery_String("Null");
 
                    //Save Url in carDao
                    carDao.addUrl(url);
 
                    //Create an istance of the CarXML
                    CarXML car = new CarXML();
                    //Set values of Car class that we want to add
                    car.setImage_url(imageUrl);
                    car.setDescription(description);
                    car.setFeatures("");
                    car.setPrice(price);
                    car.setUrl(url);
                    //Save car in carDao
                    carDao.saveCar(car);
                } catch (MalformedURLException ex) {
                    Logger.getLogger(Carhub.class.getName()).log(Level.SEVERE, null, ex);
                }
 
                //Print result to console
                System.out.println("Car Url: " + carUrl + "\n" +
                        "Image_url: " + imageUrl + "\n" +
                        "Description: " + description + "\n" + 
                        "Price: " + price + "\n"
                );
            }
 
            //Sleep for the crawl delay, which is in seconds
            try{
                sleep(1000 * crawlDelay);//Sleep is in milliseconds, so we need to multiply the crawl delay by 1000
            }
            catch(InterruptedException ex){
                System.err.println(ex.getMessage());
            }
        } 
    }
}
Price Comparison Website/Back-end/scraper/src/main/java/CarXML.java
Price Comparison...
SOLUTION.PDF

Answer To This Question Is Available To Download

Related Questions & Answers

More Questions »

Submit New Assignment

Copy and Paste Your Assignment Here