What we did

Architecture

This project was executed in a close collaboration with the Listglobally technical leaders. We used a microservice architecture and all the services were hosted in Microsoft Azure. Various storage solutions were implemented such as SQL DB, Document DB (NoSQL), table storage, blob storage etc. We worked with .NET Web API for the backend, and AngularJS and ReactJS for the frontend (in separate apps).

Import and Export

Over the last 5 years in the industry, Listglobally has built an extensive network of users in more than 50 countries which brought up a challenge: each client usually had one’s own listing format in XML, CSV or JSON and we had to adjust working with all of them. In order to handle those formats, we built a service and called it Teleport API. It can take any listing and map it to another one with visual mapping builder. It also generates a structure for every format that was used in the system so it might be reused for a purpose of currently existing formats mapping.

After the mapping is complete, it’s tested by using real data from either a client or our system depending on the direction of the transformation. The system automatically puts it in the line for processing and when a slot is available it is converted automatically by uploading the file to the cloud storage and converting it. The system can spin up a defined amount of instances for processing (up to 10 at the moment) so that we use cloud resources rationally.

When client’s data is mapped to the new format we can import it to our system by using import modules (we call it IO API). The process also works the other way around: when we send our data to a client, the file is exported in our format first and then gets converted into client’s format.

Publications

Every listing that is imported into our system can be published on one or multiple portals. We built Publishing API to serve that purpose. It takes an uploaded feed and processes every listing in it to check when it could be published.

For each client, we can set parameters and/or restrictions that will be applied to the listings such as what countries they should be available in, languages to be translated to, and data quality restrictions to be applied (for example, only listings with pictures and prices and so on).

Publications are not uploaded in real time so there is a delay between importing a listing to the system and having it published to external portals due to the technical aspects of the process. For the cases where real-time publication is required (for example if you click on a “Publish” button in the backoffice for a specific listing), we have separate endpoints that will calculate publication rules instantly and deliver it. However, this option is only available for a small number of listings, otherwise, that would make visibility matrices too large.

Search

Imported and published listings should be easily searchable so we used Azure Search service for the implementation of the Search API. It must load a massive amount of listings, do searches, sort and narrow those down (faceted search), as well as complement search results with listings info, resources and other things requested by a client (such as website, mobile app or a service).

Users and Accounts

Security was a key element of this project. Thus, we managed our users’ accounts in a way that would allow creating a structure of entities suiting the clients’ needs and providing security for their data so that no one could retrieve their listings without permission.

Leads

The purpose of any real estate business is lead generation. When the portal receives leads, it delivers them to a seller of the property or an agent. Listglobally can process the leads from emails, import them from the feeds, or receive their requests directly in the API. Depending on the settings selected by a seller, the portal can forward those to his/her email, use a third-party API or simply keep it in the system account.

Statistics

Every publication, search, and view of a listing or exporting it to a customer is an action that can be tracked through our Statistics API aimed at gathering and providing this data. We track the leads and views on the listings and, for example, use publication statistics for a user’s billing and reporting.

Scaling & Caching

Every API and web app uses two or more instances to ensure stability and SLA. For heavy duty services like images or search, we use 4-8 instances with dynamic scaling. Each API is scaled accordingly to be used by the team responsible for it. APIs that require caching their data have a cache layer implemented (in memory, Redis, etc)

The Outcome

We’ve been working on this project for 5 years now. Together with the Listglobally team, we built a service that is able to handle an impressive amount of data by being fully automated and working well autonomously. The system currently stores millions of listings for hundreds of clients, generates thousands of leads monthly, and hosts over 120 portals (full applications and white labels) with an outreach of 216+ million consumers every month.

Tech stack