OmniSci

What is OmniSci?

OmniSci is the analytics platform focused on speed. At the core is our open source SQL database that has been built from the ground up to run on GPUs, taking advantage of the massive parallel processing and fast, built-in memory to make SQL queries run 100-200 times faster than mainstream SQL databases on CPUs. On top of that we have built a rendering engine and a visualization platform that take advantage of the fast SQL core to make even the largest datasets fully interactive.

Check out our ships demo to see 12 billion rows of telematics data, queried and visualized in milliseconds, with no need for aggregation or sampling.

We’re very excited to be partnered with Uber for this challenge, using the Uber Movement Travel Times data. This public dataset includes average travel times between traffic analysis zones, and offers insight into how speeds are changing across an entire road network.

OmniSci Challenges

  1. Uber Movement Visualizer - create a custom web app, using OmniSci on the backend and Uber’s kepler.gl on the frontend, that visualizes all of the Uber Movement data. Take a look at the existing public visualizer. How would you improve it? If we wanted to make all of the data across all of the regions available in a single interface (which is possible with OmniSci), what would it look like? (See the Lidar Visualization app as a reference.)
  1. Traffic App - create a custom application (mobile or web app) that visualizes a key finding or predictive capability based on the Uber Movement data.
  1. Example 1: given a user’s location (GPS or zip code), estimate the current and future travel times to any other destination in the region.
  1. Example 2: given home and target locations, find optimal commuting times between the two destinations.  Or, given a desired target location, find optimal locations that are within a given commute time.
(See the TweetMap app as a reference.)
  1. Other Challenges - Feel free to use OmniSci with any of the other CalHack challenges, to help make data from your hack visual and interactive. We’re offering free instances of OmniSci to all CalHack participants, even if you’re not hacking one of our challenges.

Judging & Prizes

  • We will be judging each hack on four equal criteria: 
  • Technical Innovation
  • Polish (design)
  • Impact 
  • Effective use of OmniSci, Uber Movement, and kepler.gl 
  • Grand prize: each member of the team will get a pair of Apple AirPods or an Amazon gift card of equal amount. 
  • Second prize: each member of the team will receive $100 Uber credit. 
  • All OmniSci challenge participants will get the chance to have lunch and meet the engineers at our San Francisco office. We’d love to have you come by, share your experiences using OmniSci, and learn more about how we do what we do.

Getting Started

  1. Create an OmniSci Cloud Account - one person on the team should go to omnisci.com/cloud and register with your school associated .edu email.
  1. Invite the Rest of the Team - once your account is activated, click the SETTINGS link in the upper right corner of the OmniSci interface. From there, you’ll see a TEAM tab where you can invite the other members of the team. The invited users will have a role of Viewer by default, we suggest you change it to Admin to grant them full privileges.
  1. Load Uber Movement Data - To load the Uber Movement data into OmniSci:
  • Click on DATA MANAGER in OmniSci Immerse -> click Import Data
  • Click Import data from Amazon S3 -> select the S3 LINK tab
  • Uncheck Private Data checkbox
  • Click Preview and you will see a sample rows from the imported dataset with the assigned data type
  • The name of the table is auto-assigned, so click Import Table to save the dataset into the table.
  1. Explore the Data in a Dashboard (no coding required) - On the DASHBOARDS tab, click the NEW DASHBOARD button in the upper right side of the screen. Click Add Chart and select a chart type from the top of the chart editor. Helpful videos can be found on Using OmniSci Immerse page, and detailed instructions on various chart types are available on the Working with Dashboards page. You can use the dashboard to explore the Uber Movement data and see what local traffic patterns are. You can also upload other public datasets to see how those traffic patterns are influenced by construction, accidents, weather, or other events. Be creative - find as many relevant datasets as you can.
  • OmniSci API Access - OmniSci Cloud Super Admins, Admins, and Developers can each create a single read key and a single write key. To do this, select SETTINGS from the upper right-hand corner of OmniSci, and then the DEVELOPER tab. Click Create Write Key and note the API Key Name and API Key Secret.
  • OmniSci Python Connection - The code segment below shows how you will use the API key name and key secret to connect to OmniSci using the pymapd python library. We’ll use the OmniSci API key name for write access and the corresponding API key secret to initialize the connection variable. All API access goes through the same host, use2-api.mapd.cloud, which then automatically forwards the request to the correct cloud instance based on the key name. The default database name is mapd and you will connect using HTTPS port 443.
  • user_str = 'API Key Name'
  • password_str = 'API Key Secret'
  • host_str = 'use2-api.mapd.cloud'
  • dbname_str = 'mapd'
  • connection = connect(user=user_str, password=password_str, host=host_str, dbname=dbname_str, port=443, protocol='https')
 

Technical Resources

  • TweetMap Sample Code - We created a custom TweetMap application using all of the relevant OmniSci APIs (connector, charting and rendering). Read the blog post that talks about how we did it, and take a look at the source code.
  • LIDAR Visualization - We created a custom, 3D deck.gl application on a LIDAR dataset. The source code is explained in this blog post, and shows how to connect to the OmniSci backend and run SQL queries. kepler.gl is an open source tool built on deck.gl for geospatial data analysis. The github repo has examples and source code.
  • Machine Learning Demo - We’ve created a few Jupyter Notebooks that explain how to integrate OmniSci into a Machine Learning pipeline. The code makes use of the pymapd Python library to extract the desired features into a dataframe, and then pass them off to ML libraries. Refer to this blog post for more details.

Additional Resources

  • Join the CalHack Slack workplace and look for the #OmniSci channel. Our trusty tribe of engineers will be available online to answer any questions.
  • Come to our workshop on Saturday from 11am-12 noon at SCET, where we’ll walk through how to use the OmniSci APIs to build custom apps (#1 and #2 of our challenges). Come join to learn or ask questions. (We like donuts, so there will be donuts.)