Simen & Knut stop by to talk about Sanity.io - why they developed it, what it does, who it's for, and the types of projects they see people using Sanity for.
Time Jump Links
- 00:51 Topic introduction
- 03:17 How did Sanity start?
- 08:59 How does Sanity work?
- 11:37 Where is the data?
- 17:42 Sponsor: Front Conference
- 19:07 How do I build a multi-user blog with Sanity?
- 25:18 How do migrations work when schema changes?
- 32:47 How do people set up repos for front end vs data?
- 38:10 Sponsor: WooCommerce
- 39:19 Public vs private repos
- 43:29 API Options with Sanity
- 50:54 How are Write API's handled?
- 53:14 What do you see with your competitors?
- 58:13 What are people building with Sanity?
To build a kick-ass frontend, we rely on each other: Designers on developers, and developers on designers. But it’s often difficult to work together; we need to learn each other's language, understand each other’s priorities and have to be willing to make compromises that serve both.
in 2011 we came up with a conference for everyone involved from concept to implementation of a frontend, and we gave it the tagline “Where design meets technology”.
WooCommerce is WordPress plugin that brings eCommerce to your WordPress sites. It's unique in its customizability and flexability. You can use it to sell physical products, digital downloads, memberships, services, and tickets, plus offer customers lots of different ways to pay, including things like Apple Pay and Bitcoin powered by Stripe.
MANTRA: Just Build Websites!
Dave Rupert: Hey there, Shop-o-maniacs. You're listening to another episode of the ShopTalk Show. I am Dave Structured Content Rupert and with me is Chris The Database Coyier. How's it going, Chris?
Chris Coyier: I was wondering what you were going to do. I was going to be like, Dave Left Join Rupert, you know.
Dave: Yeah. [Laughter] Yeah.
Chris: Dave Relational Rupert. No. Uh… [Laughter]
Dave: Flat File Abs -- go ahead.
Chris: I've been wanting to do this forever because, by virtue of having these two wonderful guests with us this week, we get to talk about an awesome cross-section of all the stuff that we like to talk about on ShopTalk Show. One, we talk about CMSs a lot just because it comes up because so many websites are powered by them; you can't avoid them. Plus, they're fascinating. The whole landscape of CMSs is just something that I just can't avoid being interested in because it's just such a cool landscape.
Not only just CMSs, but headless CMSs, which have been a fascinating topic for years. Then how that combines with site generators and JAMstack, and all these really modern terms that aren't just hype anymore. They're starting to be really meaningful across the whole Web.
We have two gentlemen here from the company Sanity. The URL is Sanity.io, straight out of Oslo, right? [Laughter] I think. We have Simen and Knut here from there. Hey, guys. How ya doin'?
Simen Svale Skogsrud: Hey. We're doing good in the heatwave in Oslo.
Chris: Is it?
Knut Melvaer: Yeah, everything is great. Yeah. Not warm at all.
Dave: How many degrees Centigrade is that because our American listeners won't understand it, but how many is that?
Simen: That's 19 degrees right now. I think people are dying. No, I'm overexaggerating.
Simen: We are used to it being pretty cold. Now it's 25?
Simen: Okay, so people are dying. Yeah.
Knut: Is that like 400 Fahrenheit or something?
Simen: Probably. Probably it's like if you're doing a slow cooking the steak, that's how you do it, right?
Dave: Yeah. It's like 82 degrees.
Chris: Captain Google says 82, which I'm sure for Norway is something like--
Simen: Yeah, so we don't do air conditioning here at all.
Chris: Oh! That's why, yeah.
Dave: Yeah, that's hot.
Chris: When I came to London, that was a problem too, right? It's like a whole city without air conditioning.
Simen: Yeah, it's like … never figured out that it's cold there sometimes and we never figured out it's hot here sometimes, so we just don't do that.
Knut: Yeah, we missed you in London, Chris.
Chris: I know. That's sad. In fact, I met both of you in San Francisco, something like a year ago, I believe at the very first JAMstack Conf. Is that right?
Simen: That's right. That's right.
Chris: Yeah, and you've been going to them since because I think the product that you work on, Sanity, which of course we'll get to in this show, is kind of highly related to the JAMstack audience, you might say. I think we'll get there because we haven't explained what it is, why it is, or anything yet. Maybe, do you want to give me a little background, Simen, on yourself and just a quick backstory of yourself and where you're at now?
Simen: Yeah, so I started out with television. I worked in television in the '90s. I'm old. But I've always been a programmer, even since my childhood. I kind of came back to that with Web development in the early 2000s. I've been running this kind of technology consultancy for years making websites, a lot of social media. We were kind of early in the game before Facebook in Norway, introducing these kind of social websites. That was our game for a long time creating these sites for companies for user engagement and stuff like that. We've also been doing a lot of rich media stuff like museum installations and audio/visual things.
Then I've always been dreaming of having a product company but never having the kind of economy, time, or often the opportunity to do it. But then as part of being a consultancy and doing other content things, we started making our internal tool for content for reasons we are getting back to. Then, suddenly, we realized this product doesn't actually exist. When you have an idea and you don't immediately make a company, we have this kind of rule that you should not get angry when the product actually appears because someone has the same idea as you at the same time.
This time, it took years and the product that we wanted didn't appear, so we then decided to become that company. Now, we aren't consultants anymore. We don't have any clients anymore except, of course, users of Sanity.
Simen: Now, we have been living the dream of being a product company.
Chris: That's nice because you can kind of build the product. It's not like you forgot what it's like to have clients. [Laughter]
Simen: It's not so long ago, you know. [Laughter]
Simen: I think it's just one year ago, we kind of, let's say, phased out the last client. Yeah.
Chris: It's a CMS, right? Do you avoid using the word CMS?
Simen: I think we kind of finally figured out that we can call it a platform for structured content. Then all the different things fit in. Of course, most people who use Sanity right now are using the CMS aspect of it.
Chris: Right. Right. Okay. We'll get more into that, for sure. Knut, do you want to introduce yourself?
Knut: Yeah, sure. I almost got a doctorate in the study of religions, actually, before pivoting into the tech as a consultant in an agency called--
Dave: Weird flex, but cool.
Knut: Yeah, so I worked for a couple years in a consultancy called Netlife with salvaging websites or, like, confronting legacy CMSs, thinking about the user experience.
Chris: Salvaging is a cool word.
Knut: Getting content out of them, right, and into something else. I did that for a couple of years. Then I got in on the early beta version of Sanity. I used that to build the new website for this consultancy called Netlife, fell in love, and I got the opportunity to work here as a developer advocate. So, now I'm doing that.
Chris: Hey, that's nice.
Simen: That's actually a good story because you were able to explain. The way we decided was Evens, my co-founder, Even's wife told Even, "I talked to this interesting guy called Knut, and he could not stop talking about Sanity and he was actually able to explain to me what you do at work for the first time."
Simen: Then we asked you nicely if he would please join us and help us explain what we are doing.
Chris: That's great. I have a story. Years ago, I was kind of a superfan of WuFoo and just would write about it and talk about it and gush about it because I thought it was a really great product. It still exists today like a form builder thing. I think, by doing that, I just kind of got their attention and, ultimately, spoke at the same conference as one of the founders, Kevin Hale. Through virtue of that meeting, I eventually started working there. I think that can happen, going from superfan, a vocal superfan, to employee.
Simen: Yeah. Really, of course that group of superfans that are around something you are making is really important, right? You're using that energy to motivate yourself. But then, of course, those people are incredible, also. Then trying to tap into that.
Lots of the people who work at our company were superfans of something we made earlier, actually.
Simen: Then we were lucky to be able to hire them.
Chris: Yeah. Yeah. There's some kind of like marketing legend, like 100 true fans or something. You can pull something off if there are enough people that really believe in what you're doing. I'm sure you're well past that now.
Simen: That's the current we need. Yeah.
Chris: Yeah. All right, so there's plenty of stuff to talk about, about the whole landscape of the thing, but let's dig into Sanity a little bit first because I think that will set the stage of where you two are at. If we're not calling it a CMS, even though I guess people wouldn't be too off base for thinking of it in that way, a platform for structured content, but it's something that you can NPM install, which is not true of most things in this genre, I guess.
You think of a lot of CMSs as PHP based or something that is just totally cloud based, so you just use it in their cloud. But this is NPM installed at Sanity or whatever.
Simen: Yeah, so it's self-hosted in terms of the user experience. What was really important to us when we made this as consultants because, of course, it was made when we were not thinking of it as a product but as something that we just needed. We wanted to have these kind of cloud-based data store like a database that is specifically for content. But then, also, we did not want this, like, one size fits all online kind of form builder experience that some of these modern CMSs are because, as consultants, we think that the editing experience is as important a part of the product that we give to our clients as the front-end itself. We really want the editors and the people who interact with the content in the companies that we work for would feel that as a delightful experience, would have an easy time to create the content in the right way.
Sometimes, we would get these weird requests like, "We want autocomplete for the architects in our company." Then if one of the partners is a match, those should be on top even though they aren't the best match; it's just kind of a respect thing.
Then we needed to be able to -- I don't think, as a consultant, no isn't a good response. You can give it a high price or something, but it should always be yes, it's hard, or yes, it's easy. Hopefully, it's easy. That's what we want. That's why it's NPM install. It's still super simple to get started, but at least you have everything locally. It's a completely … React app.
Chris: What is it that you're pulling? You're pulling an admin interface for the data?
Simen: Yeah, so that thing that you NPM install is definitely a CMS. That's the CMS part of Sanity. What people sometimes get confused about is that the content is never on your laptop even though the entire experience is running on your laptop because our APIs and content stores are always cloud-based.
Chris: Yeah. That did confuse me at one point. The data is not like a Git LFS thing or whatever, or it's not flat files. It's not -- it goes to your cloud.
Dave: SQLite or something.
Chris: SQL Lite, right. It's not like that. Your data lives in your cloud.
Simen: Exactly, so we make, basically, a JSON database optimized for being a content store. One of the reasons for this is that, of course, a very important part of this is that you should be able to collaborate on the content, so people should be able to work at the same time seeing the same things. This, out of the box, works and looks like Google Docs in the sense that you will see other people's edits as you edit.
Chris: Oh, really? It's real time-y, too? That's cool.
Simen: It's completely real time-y and everything you do becomes like a small little patch that gets distributed to everyone who is looking at the same thing as you. Have you seen the Gatsby Preview feature where you can actually live preview as you edit? Because Sanity is built around real time, that was really -- implementing that was really beautiful because sub second, as you edit your stuff, you will immediately see it reflected in the previews from Gatsby Preview. That's kind of part of the joy of having all this be on the cloud, cloud-cloud-cloud.
Simen: But then what you have is APIs. Of course, we have GraphQL or queries where you can interrogate your content as a database because that was an important part for us. Content is nice thinking of it as content, as running text, as images and stuff, but also sometimes lots of the content is these kinds of fields like authors or years. Then you should really be able to query those like it's a database, so that's why we wanted our APIs to be also like a proper database.
Chris: Yeah. I have lots of questions about that because that's fascinating to me. Let's say you're going to go with a flat file system. I just need to keep it simple, like keep my head wrapped around it. I have this website that I run on 11ty, which is just like a flat, node based, static site generator kind of thing. I was like, I have all the options in the world. The data is so simple. It's just conferences. That's it. There's no authors. There's no years, really. There's kind of the date that the conference happens, but the date is about as simple as it gets.
I wasn't sure that I needed a real DB. I was like, this feels kind of perfect for flat files, just Markdown files. Each conference is a Markdown file, period. I feel like a Jekyll site with blog posts can kind of be like that too. It's not that complicated. It's just a blog post. Just make a Markdown file and throw it in a folder.
Okay. Fine. I went that route, and I don't really regret it. But that's not the most complicated structure of content. The second you get a little bit more complicated, like Dave's blog is a Jekyll site, right? Dave wants to blog. He fires up a Markdown file, writes the blog post, commits it, runs Jekyll or whatever.
Let's say that me and Dave want to start a blog and we both are separate authors. I feel like, immediately, it breaks down a little bit because you wouldn't write a blog post and say, "Author: Chris Coyier," because that's not how relational databases work. When you store a blog post, the author should be the ID of an author in a different table and there's an author's table. That's the concept of a relational database, right? I feel like the flat file system breaks down the second you need some kind of relational something. Does that make sense or do I have it--?
Simen: No, I totally agree. I use plenty of flat files, Markdown, and YAML files in my time. I love the simplicity of it for personal projects. One of the goals that we had here was also that the simplicity of just, let's say, when you start a complex site that you're going to spend a year making, you will often start by mocking some content out. You will probably make a JSON file or something and just stick it in there.
Then when we had Sanity for the first time, being a simple thing to start, I realized, "Oh, finally, now it's as easy to make a simple database in Sanity as it is to make a flat file in JSON. Then I can just start.
Simen: The I don't have to backtrack when I'm hitting my head against some limitations. There are two kinds of limitations here. The one thing is that you're talking about the data model like the complexity. It very quickly can break down, as you say, like you have some relations or let's say you have to have some kind of Markup in the text describing your conferences. You want the program to look differently and know what are events and stuff like that.
Simen: You have to invent secret codes in Markdown or something. These things make this a system. One thing is that there's no kind of correct way of doing that, so you end up making these secret codes. Then that becomes your CMS that only you can use. Now, maybe you want some stringer to help you maintain this. Now you have to train someone whereas if you had a simple way to get this up and running in a CMS that was simple. I would kind of put that using Sanity in a simple way is both free and very simple, as simple as flat files.
The only concept you need to understand in terms of the database of Sanity is that it is a flat list of JSON documents. There is nothing else. It's just a bunch of JSON documents. The thing we do on the kind of backend is indexing those in clever ways so that when you ask a complex question like, "Can you please give me all the blog posts and then just link in the names of the authors from these other kind of JSON documents," then we know how to do that very quickly. To you, it just looks like a bunch of JSON documents.
Chris: This episode of ShopTalk Show is brought to you in part by Front Conference, literally a conference. A great name for a conference, Front Conference, and URL. It's literally FrontConference.com. It's coming up August 29th and 30th in Zurich. I've never been to Zurich but, guess what, I'm going to now. First time ever in Switzerland because I'll be at Front Conference and I can't wait. It's going to be awesome.
Of course, the first day, like many conferences, is workshops, so check those out: Harry Roberts, Andy Budd, some great people teaching great stuff. Then a full day of talks the next day. There are a ton of speakers there, people I'm really looking forward to meeting; not the normal names, actually, of conferences that we've had. I feel like there are a lot of fresh voices and a lot of really strong, traditional -- I don't know why I would call them traditional speakers but, of the conferences I go to, familiar names, a good mix of both. I absolutely can't wait.
Just go to the URL and see if it's for you, FrontConference.com. If you live in Zurich, I wouldn't miss it. If you live in Switzerland or anywhere over there in Europe, I assume it's a lot easier for you to get to, so get your butt there. I can't wait to meet you there, or you could come over from the States and hang out with me and we'll drink PBRs at some pub. Oh, I'm just kidding. We won't do that. We'll have fun in some other way.
You know what? If you absolutely can't make it, this is cool of them, too. They're recording all the talks, but they're going to live stream it as well. Mark it on your calendar and follow them on Twitter. If you want to just see what people are talking about at this fantastic conference, you just can, for free, by doing the streaming thing. Pretty rad of them. Check it out.
Chris: Let's do the blog thing. Me and Dave want to start a new blog and we're going to publish blog posts, but some of them are by Dave and some of them are by me. I NPM install Sanity. What's my next step? How do I model that data? Is there a learning curve? Is there a schema that I edit? What's the next step?
Knut: Well, this is a perfect question for a developer advocate, right? [Laughter] Actually, now you don't even have to NPM install anything, at least not the first time, because you can now go to, like, Sanity.io/create and there you will find this. I think they used to call them wizards, right?
Simen: In my time.
Knut: It's like, I think, a couple clicks and then we will put all the code for the project in your GitHub and deploy a Gatsby and Sanity … rig for you on Netlify and you have started with a blog. Then you can tweak the code and make it your own. That's a way to do it, right?
Simen: Yeah, that's exactly right. Actually, we made this conscious choice of making the schemas be something that looks like code. It's basically JSON documents. You have to read a bit of documentation. Actually, your specific example, you can basically use one of our starter examples and you'll get exactly that, but let's pretend that doesn't exist.
Chris: Oh, I see. You're describing the fields of a blog post because that is part of your schema. You don't get some standard, like you install WordPress. You open up WordPress admin; there's at least a basic set of fields already there sitting there waiting for you, but that's limiting in a way.
Simen: Yeah, the legacy.
Simen: Yeah, because it used to be like a blog system in its early days.
Simen: But, of course, it's easier to start with something. Usually, we recommend that people start with one of our default schemas. One of them is a blog.
Chris: yeah. Then we can see the syntax of the schema, right? And be like, "Oh, I see. There's a title and a big text area kind of thing," but I don't actually need the text area. What I need is an image uploader and I need a date field and a color picker or something.
Simen: Exactly. Then to us, we got a lot of questions in the beginning, like, why don't you make a visual schema builder or something like that? We thought that, to us at least, the data model should be treated by someone who kind of has the programmer approach. They might be marketers, but marketers who know how to program, because then you need to kind of--
Simen: …think with different fonts and stuff like that. Then if you are the kind of person who knows how to and likes to program, you also want to be able to paste an example. Let's say you ask me, "How do I make a grid of images?" Now I can paste here an example and you can have it in Git and you can have it on the revision control. You can merge different schemas. That's kind of having it as code and text is very useful. Then, of course, on the other hand--
Chris: Yeah, right. Well, if you're making a product, you've decided this isn't for -- the marketing for this product isn't going to be, "Build a website with no coding!" You've intentionally said, "No, there are some parts of this that we could do that way. We're intentionally choosing to keep it a coding enterprise because the results are going to be better," or, like, "Trust us."
Simen: Yeah. Exactly. We say, "For programmers, this is the best way to then delight your content people," because we spend most of our time making sure that the editing experience is awesome for those guys and the programmers can kind of easily design the content model the way they feel. One my dreams was that the data model should kind of feel programmer-y. It should feel clean and direct.
Simen: You shouldn't make too many concessions based on what goes on which page and stuff like that. You should kind of think about it in principles how do I model the content most correctly and structured. Then I still want my editors to get something that feels obvious to them and where everything is named like they name things inside the organization, for example.
The goal then is that you should be able to then when you have a major blog with Sanity, you should be, when you add one more person to your blog, that should be like just adding that person to the project and there should be no training. Every secret way to add YouTube embed or something should be obvious and be a button and self-documenting, so that's the dream. I think we are….
Chris: So, when you're building a schema, part of it is like, "Here is what a blog post looks like. It's got a title, a date, a description, and all this stuff." Is there also a schema for, "This is what an author looks like," and they are two different schemas?
Simen: Exactly. You would make a new document type, let's say an author.
Simen: Probably have a bio or something. Then you'd say on your blog post, you'd say, like, authors. Probably you should make it a list.
Simen: You say these are authors. It's like an array or references to authors.
Simen: Now you immediately have a user interface where you can search authors and you can add authors and all that niceness.
Chris: Hey, that's nice. Right there is a cool feature because I have WordPress sites and I'm largely a fan of what it's done for me over the years. But just right there, when you assign an author in WordPress, it's not a list. It's a dropdown menu.
Simen: I've made that mistake like seven times in my career, talking with a singular author and then realizing, "Oh, of course." Even the one when you create a new document. It's always you that is the author and you can't change it, so yeah.
Chris: Oh, yeah.
Simen: Yeah, yeah.
Dave: The immediate thing coming into my brain is, how do migrations work? Let's say the posts actually need another image and it has to be smaller than the main image. Then on the bio, we're not doing Twitter anymore. We taking that out because Twitter is a terrible company. Then on the homepage content, we're not calling it "Title." We're calling it "Hero Lead" or something.
Chris: Yeah, so you change the schema in Git Committed or whatever. Oh, how is real time work now? I'm looking at that page and Dave just changed the schema on me.
Simen: Yeah. You'll get some yellow. If you have different schemas, you will get some yellow alerts alerting you to the fact that there is content here that doesn't make sense in the current schema. If you are brave, you can try to resolve them yourself. Usually, of course, you have exactly like you would have on Git. You can have your own private copy or data set. As a developer, you would probably create a copy and then mess with that.
Chris: Oh, I see.
Simen: Like the development team for one feature would have their own data sets where you can….
Chris: Cool. That's kind of like an offline mode, too, in a way? I can continue to work….
Simen: No, it's not offline. It's more like a branch in GitHub. It's just a separate--
Chris: Oh, I see. I see.
Simen: It's still in the cloud, but it's just like this specific dataset is for experimenting with the new author structure. Say we really wanted to double down because we want to make separate pages, so we load lots of content on the authors. Now we can play around with that and we can make our scripts to rebuild content or add.
Of course, adding fields is very simple. You just add them. Deleting fields is similarly trivial.
Simen: Of course, when you made that mistake where you have a single author and then you realized that you needed two, then you need to write a small script that will change those fields. It's not very hard, but that's one of the things where we maybe could be doing an even better job and that could be even more convenient.
Knut: We have some … migration scripts and so on that also do it real time.
Simen: Some of these are so obvious that we should probably have a tool for it, but we will make that. We'll get around to it.
Dave: Yeah. For me, it's clients. You maybe experienced clients sometimes change their minds, and so you're always trying to keep up with their demands.
Simen: Yeah. That's one of the kind of sticky notes that went up really early to us. It was like we didn't want any disruptive updates. We wanted a smooth upgrade path. Then we also wanted there to always be a way forward when the client comes with this kind of super disruptive request. That's one of our dreams. I hope we are delivering on it.
I used to work with television and I worked in print. I've also been a writer for some time. Then, in this field, when you want to try something new, you just try it. We need a new segment in a TV show; we just make it a couple of times and see if people like it. There is no, like, $500,000 weight for the developers to make a new API and then first you have to kind of integrate with providers and whatever. Before you actually can test whether your idea is any good, you've already spent the budget.
What our dream was with Sanity was, if you need something new in your site like your appliance manufacturer has this idea of having a celebrity endorsement or something, they shouldn't then have to spend lots of development hours in trying it out because you just add a number of new schema sections and then you try it out. The API just writes itself. Then, if it works out, nice. Now you have the basis of it and you can keep developing. If it didn't work out, you can just delete the whole thing and you didn't spend a lot of money on it. That's kind of the dream of this to be able to continue to try things and making sure these things are pretty cheap.
Dave: Yeah. I think I like that because, to add a field to the database, you're kind of just editing some JSON, more or less.
Dave: I don't want to oversimplify it.
Simen: Actually, you just do that.
Dave: Okay. You just said, edit some JSON.
Simen: You just claim. You just claim it has a title field and claim it has an image field. Now it has.
Dave: I think of WordPress development and I don't want to make huge comparisons, but something like advanced custom fields is very popular. We like it here. We use it.
You have to kind of go through the admin section, add your fields, and then you go to your posts. Then you lay out those things. Then you have to go through every post and add in content. It just seems like you've kind of eliminated a lot of the GUI in lieu of JSON. You can kind of develop and make content experiments faster.
Knut: Let's say you have a file with some JSON documents. You can actually just import that file right into the back-end, start querying it without doing anything in the editing experience. That can come later.
Simen: That's actually why we need to call it a structured content platform because some people aren't actually using the CMS at all.
Simen: One of the first successes we had was a TV streaming company came in here. They had a problem. Like a cable company, basically. They had a problem that they couldn't provide a list of upcoming movies across all channels because their APIs were made by someone who talked in terms of channels, so they had to make one request for each channel from the front-end.
Then we could show them, within a half an hour. Instead of having the pitching meeting, we just did the work for them. We just piped in their program data into Sanity, and now they had this kind of powerful query API. It was already JSON. We didn't have to make anything.
Then later, we could add the schema and make it editable in the CMS. For that use case, they didn't actually need it. Before they left, they could actually show.
Then later, actually, it was a bit funny because this was really early. Then we called them later to ask them, "Are you going to use this or are you going to be a customer?" They were like, "Oh, no. We already deployed. We're in production." I was like, "Okay. Our free plan is maybe a bit too generous right now."
Chris: That's funny. That's funny. It's kind of like if you don't use the CMS, it's like queryable JSON in the sky.
Simen: Yeah. That's a good way to put it. We should probably add it on the front page.
Chris: [Laughter] Let's say I don't care about -- or here's my list of things I care about for a project, just a theoretical thing. I care about structured data and I care about APIs to get access to that data. That's it, kind of. I have this idea for a movie review app or something. I need very little. I'm thinking headless CMS-y here like that's all I want, and so I install Sanity, get a little schema ready and cool, and then the APIs that I hit, is it like API.Sanity.io/whatever? I'm hitting your URLs for my requests for my data, right?
Chris: Okay. I can use Gatsby, sure. Let's say I want to use something else. I'm just going to build a React site and just hit your APIs as it loads just because that's what I feel like doing on my team or whatever. I can do that, right?
How do most people set it up? Do I have one repo that's my front-end and a separate repo that's the data management repo that deals with Sanity stuff? They're two separate worlds, which feels like the…?
Simen: That's how I used to do it myself. That was probably because the first project we did of this size was this architecture agency OMA. That was kind of the perfect storm because they wanted a website and we made exactly what you are saying. It was the coolest thing at the time. We didn't do static site builds at the time. We did really fast, single page apps like isomorphic apps.
Then the whole idea was like we have fast APIs. We cache things. You are able to combine loads of queries, so we can kind of query for everything in one request, get everything back, so we don't have to go, like, the worst part of these APIs that I used to use where you'd have to hit the server like 50 times for a page load. Then we fixed all of those things.
Then those guys wanted to make books based on the same content, and we had said that in a pitch meeting that if you do structured content, then you make books from the same content with no content rework, but we didn't really know that. We just said that because we'd never done that. But they immediately ordered the books, so then we had to do that. That worked out really nicely. We made books.
Then we had talked to the business development people there. They had two people. Those guys have been running since 1979. A really legendary architecture agency.
Two guys were tasked with just remembering everything. When the Amir of Dubai or something came to order some kind of insane retail empire that's also a bridge or something, they would be the ones who would remember what have we done that looks a bit like this in the past and then make this kind of list of projects and show it to them.
Then we had kind of told them. If you do structured content, we can make advanced searches and stuff. Then they came back and we're like, we want our searches because those kinds can be more useful things than just remembering things.
Then we made this super complicated search, too. We didn't have to do anything with the content because now we knew where the buildings were. We knew all the kind of milestones in time. We knew every architect.
Chris: Interesting. You don't even need Algolia or something with Sanity.
Simen: Exactly right because we have a curatable database. You talked about that day that you want to be able to change things because the client always comes up with new needs. As long as it's simple enough to get started making everything proper like making sure every piece of content is actually coded, so we know what it means. This came from WordPress, so it was like this heap of text documents where it created architects, locations. Everything was inside the text, so we couldn't actually curate that in any way. The only way that content could be used for was making that specific site.
When we then got all of this structured, meaning that it was all inter-schemas, the content people now did the same kind of work in a simpler tool. It's easier to not make mistakes and remember to put everything in. Then, without them even knowing it, it becomes books and it becomes the business development tool. It's just fields from the same work they already did.
Coming back to your question about repositories, that experience made me kind of always make the content editing thing its own thing because I never know how many other things. There might be an iOS app or whatever. It might be a different team, so I kind of like to keep them separate, but I see most people actually make a repo, like a super repo with the front-end and the CMS in the same repository.
Chris: I could see it work both ways.
Simen: If you're thinking, if you're coming from the WordPress kind of world where you have one front-end, you're thinking in terms of one front-end and one back-end, that's the easier thing. Of course, then still suddenly someone comes and wants an iOS app or something.
Simen: Then you have to refactor that.
Chris: It probably wouldn't be too hard to split the repo.
Simen: [Laughter] You're right. That's probably going to be one of the easier tasks.
Dave: Yeah, that's got me thinking. I like having my folder full of Markdown files for my Jekyll site, but it isn't super portable. I guess I can port it to another static site generator.
Chris: It's a good point.
Dave: But it's not like--I don't know--build a watch app for my blog or whatever. [Laughter] Not that I would.
Chris Enns: This episode is brought to you in part by WooCommerce. Today, I want to tell you about three exciting updates to WooCommerce. The first one is that WooCommerce version 3.6 shipped recently and added product blocks, so you can build rich pages with blocks in sites running WordPress version 5 or higher that show products by category, best-selling products, handpicked products, newest products, and products with specific attributes, and more. Super handy if you're using the new Guttenberg WordPress editor.
The second cool WooCommerce update is the new WooCommerce mobile app for iOS and Android. You can track your store to see which products are performing best, check revenue, view and manage orders, and get real time alerts notifying you about store activity like new orders or reviews. Be sure to search for WooCommerce mobile app on the iOS app store or Google Play today.
Finally, if you're running a WooCommerce site right now, you should check out the new dashboard for WooCommerce admin that the team released as a feature plugin. It's what's going to eventually be the default dashboard in WooCommerce, but you can check it out right now. You get all sorts of data and analytics on your WooCommerce store with a completely customizable dashboard, new reports, and a new activity panel to alert you to what's happening right now in your WooCommerce store.
If you'd like to see a WooCommerce store in operation, go check out the CodePen shop at blog.codepen.io/shop and try ordering yourself a shirt. The CodePen shop is running on WooCommerce and the shirt is one of my favorite shirts that I wear each week.
Our thanks to WooCommerce for sponsoring this episode of ShopTalk Show.
Chris: You know what I'm fascinated in? I don't know if you have a story for this. I find that there are very few stories for this, so consider it a feature request, I guess. Because, for example, Dave's blog is on Jekyll, it's probably not a public repo. Right, Dave? You probably keep it private, right? It'd be annoying to get pull requests for grammar and stuff.
Dave: You know I got that lock icon.
Dave: Oh, yeah.
Chris: Heck, yeah. Let's say you didn't want that. My conferences site is not a private repo. It's public on purpose because what I want is PRs for content and I get them, a lot of them, which is great. The flat file system works for that.
What is the story for PRs for content in the headless CMS world?
Simen: It's a very good question. We actually have a client who are going to build something like that with Sanity right now, so we need to get on top of that very quickly. It's exactly like you said. Because it's already real time and shared, and we already have -- this is not something most people use at this time, but we have very advanced access control error systems in the back-end, so you can make really specific rules like saying you can only edit your own contributions or you can only edit them, but you can't actually publish them. You can make these kinds of rules. What we don't have right now is guest users, users that could come in, for example, only proposing content and stuff like that.
Simen: it's pretty easy to add and it's definitely something, so these guys are making a platform that is used for teaching programming to kids and they need schools and people who provide the accessories to the system to be able to kind of go in and add their own content, edit, and manage their own kind of subsections of a site in Sanity. Then we have, for the really insane projects, the ability to just turn over user management to your own systems. You can just take over, just disable all of ours, and then you can actually implement this yourself. But of course, that's for someone who is pretty big that have lots of resources. Then I guess what we will do is, based on these experiences, we will learn what features are going to be common. Then we can add them to the generic offer.
Knut: You know I implemented single sign-on in one day, right?
Simen: You did that. You thought it was hard.
Knut: It is not.
Simen: Of course, the hard part is then you have to do all the rest. If you want to have guest users, then you have to control which access rules apply to them and stuff like that. Of course, for a normal programmer who has a couple of weeks to spend on that, that's completely doable.
Knut: Like in the spirit of JAMstack, you can also use something like … like that. Yeah, there are services that do stuff for you as well. Yeah.
Simen: I'm probably exaggerating the complexity of it. I just think all of those things are always so -- there are so many corner cases like, oh, your guest users are now able to do something you didn't imagine and stuff like that. You at least need to be careful. But I agree; it's a super interesting use case and it's also one of the best reasons to actually use flat files. It's something we should really get on top of.
Chris: Perhaps. I was just curious because it's a use case that I think of a lot that's like, can somebody else edit this? Let's say I have a grammar problem on a blog post. I don't even want to hear about it. Just hit the edit button, fix the grammar thing, hit the button. It becomes a PR. I look at it and say thank you and it's done, like a fricken' wiki, you know.
Simen: It's interesting because it's actually where we came from. The back-end that we used for the first version of Sanity was basically a set of microservices that we used for social media, the things we'd made in social media. Basically, what we always did was having systems where people, random people, could come in and contribute things and comment on things, sort of vote for things. What we didn't have was a CMS where you can kind of author things by authorized individuals. That was the new thing we added.
Simen: Then we deleted all the other things, so we don't have them anymore, but we should probably put them back.
Chris: Let's talk APIs for a minute. It's not REST, I would think, because that would be weird because Bespoke Schema for every set. I guess you could programmatically create REST routes, but it just kind of seems like that would be weird and not particularly useful for this new world.
It seems to me that, because you talk about Gatsby a lot, it seems like a natural fit with Gatsby, which makes a lot of sense. Gatsby is very GraphQL heavy, but it's kind of a weird GraphQL. I don't know. It's like GraphQL once it arrives at Gatsby or something. I don't totally understand it. It doesn't matter, right? You just connect them and then you use Gatsby's weird GraphQL or whatever.
Let's ignore Gatsby for a minute. I just care about your own APIs because I just want to BYO front-end. I just want to do whatever I want. I want to build a Vue app, or I want to build a Ruby on Rails app or something, but just use your APIs for the content in it. It seems like you support GraphQL kind of, but you have stronger feelings about your own version of an API and this Grok language. Can you tell me about that?
Simen: Yeah. Probably, because of … technical reasons, we really wanted to use GraphQL because it seemed to be something that everyone wanted to use. When we looked into it, it turns out GraphQL isn't really -- I mean it isn't a query language, right? GraphQL is a very, very beautiful API pattern. If you want to craft a very specific API for a specific piece of structure of content, GraphQL is really great. If you don't know the structure like … like if you don't already know the structure of your content, as a general query language it becomes pretty ugly. It's kind of just rubbing GraphQL the wrong way.
Dave: Can I give an example? You would need to know posts have authors in GraphQL to get an author. You're saying you can just say, "Give me the author." Is there an example?
Simen: Let's say it was important to you how many authors there were. I want to be able to ask, like, "Give me all posts that are written by more than one author," where Chris is the second author. Now, in GraphQL, you would create specific filters and specific functions to do these specific things. What GraphQL is really awesome at is being able to control exactly what your API consumers can do with your content, which is exactly the opposite thing we needed because we needed a way for people to be able to put in a bunch of JSON documents and then query them like a database without running through hoops.
People who support GraphQL as a general query language have done is basically kind of encoding every kind of different kind of filter or constraints in terms of GraphQL, which I would put is not really using GraphQL in….
Chris: That's kind of a good point. You don't just NPM install GraphQL and then you have an API. You have to teach it all kinds of stuff about what you want.
Simen: Exactly. That's how you get the beautiful GraphQL API.
Simen: That's what we wanted. Then you have to have something to implement your GraphQL API in terms of, how do you express them? Let's say you implement this kind of multiple authors filter. Then how do you express it in the kind of level below? How do you then ask the service, in this case Sanity, about this? Then this is Grok.
It's basically something like SQL, but it's for JSON because JSON has a different -- like SQL is always interrogating tables like rectangular tables, the rows and columns, whereas JSON has this kind of tree structure you can have like arrays inside objects. It's a different kind of beast. Grok is just an attempt to make the simplest query language where you can interrogate JSON on its own terms and then make sure it's the best way. Actually, we tried to make it look like GraphQL and we tried to make it be the best way. Actually, for example, our own GraphQL mapper basically is just a text transformation. You just translate the text of a GraphQL mapper.
Chris: Oh, funny. Really?
Simen: One Grok reader and then it fires the other way.
Chris: That's kind of cool, but does that mean it's limited? You can't do these fancy--?
Simen: Yeah, right. That's because of popular request, we also made this kind of generic GraphQL API, but we don't think it's a very good thing to do. We try to get people to go and kind of craft their own GraphQL APIs. In the future, we'll probably be able to provide some tooling for that. The whole idea is that GraphQL APIs should be handcrafted and carefully thought out whereas Grok is a general query language.
Chris: Yeah. Yeah, so if you really want to use GraphQL in Sanity, you can, but the best way to do that is for you to get personally involved and make your own schemas or whatever, like set up GraphQL yourself.
Chris: Don't just say, "Sanity, do it for me. Give me the best API ever," because you just can't.
Simen: [Laughter] Yes. Also, very likely you have other services as well. Sanity is the best thing in the world but, usually, people have also other things that they want to query. It's very beautiful to get all that into the same GraphQL API so that a GraphQL mapper is really a good place to kind of control. There, you can make sure all your front-end see the same thing.
Chris: Sure. Sure, so if I want to ask for the weather, too, I can make that part of my GraphQL API and it doesn't hit Sanity for the weather; it hits some other API.
Simen: Exactly. Maybe you want to keep your flat files for the user contributed content and then you have some other content for the same site that you want to kind of have the power of Sanity for. Then you can kind of stitch them together and provide one GraphQL API. If you have complex API needs, that's how you should do that.
Then if you have something simple, if you want to work fast and get results quick, I think Grok is made to power your site directly. That's how I have personally used it because one of the things we really wanted, which is shared by GraphQL, is this ability to ask for all the things you need in one goal and not to fetch anything you don't need and to be able to kind of include any related content in the first request. Grok is really awesome at that. I wouldn't be afraid. As long as you kind of develop the front-end and schema together, it's super powerful to just go bareback without GraphQL in between. Then when you do static site builds, it's a very different story because then what you need is to have all your content and get it just really fast.
Like you said, Gatsby has its own kind of API internal. It's running inside the builder while it's building. What Gatsby needs is basically just to get all the documents really quickly, so we have a separate API for that, which just basically streams all the documents to you as fast as possible. That's why the Gatsby build in Sanity is really fast because that API just kind of disables all this fancy logic and just gives you the raw data really fast.
Chris: Hmm. While we're on the subject of APIs, quick -- well, there's some more stuff I want to ask. What about write APIs? Is it a one way street? Are these only read APIs?
Simen: Oh, yeah, so we have a separate set of writing APIs. As I said, we have a strong focus on collaboration and we have worked with lots of newspapers in the past using, for the sake of being a bit pitchy, let's call them legacy CMSs, like the big media-based CMSs. [Laughter]
Chris: Sure. Let's call them legacy CMSs.
Simen: Yeah, so this a huge Java thing, really powerful and have a long history in media. Then it turns out, if someone is editing something, like a journalist is leaking a story, of course, they also want to get in on the kind of AI and real time stuff, so they have these bots that will record. Let's say there's a football match and there's scoring. Then it will go in and update all the refences of a match and add this kind of new result.
Simen: Then it turned out they couldn't because, if a story was locked, this bot would have to wait and come back later. We saw all these problems and then we realized that we need the writers to be able to be as real time and meshed with the users as--
Chris: So, if somebody has the story open and they're editing it, there's a concept of locked in your API, right?
Simen: No, so we don't do locking at all. What you can do with Sanity is you make a patch. You'll say, "In this paragraph here, I want to do this change," and then that will be changed exactly like if a human went there if it was like Google Docs.
Simen: You can actually merge change and you can have the write APIs come in as humans. What we have is a patch API where you can describe these changes or you can do it simpler and you can just say, "Create these 700 documents." But then the simplest way to import things is just having a bunch of JSON documents. You can use our Sanity import tool, which will then just stream all the data into Sanity. Knut wants to say something.
Knut: [Laughter] Yes. Having the CMS part open source, that means that all the APIs that the CMS has used, you will have access to those too. You're always the API first class citizen.
Simen: We don't have any private APIs, actually.
Chris: Hmm. I'm sure this isn't your favorite kind of thing, but there are competitors in this space, maybe none that do exactly what you do, but kind of. A big player being Contentful who will probably also show up at JAMstack confs and stuff too. It looks like their API is RESTful, I guess, and they have GraphQL.
They don't use Grok. That's your own kind of thing. Maybe it would be cool if they did someday, but what do you see when you see that then? They're like, okay, so they do offer a GraphQL API. Is it just super limited or did they make conceits that you see as incorrect or something? Do you watch other competitors? What do you think about competitors that do offer it?
Simen: Yeah, so this is, of course, really early days. We try to not focus too much on these other similar services.
Simen: The cake is so huge now, we kind of compete more with these, let's call them, legacy CMSs.
Simen: There's so much business, but still, of course, there is a reason that we didn't want to, as consultants, use Contentful. One of them being this thing where, at that point, all of the other cloud-based CMSs had closed source editing experiences. It was impossible at first to actually change it in a meaningful way.
Chris: Oh, sure.
Simen: That's where we started. Then, when we looked into the APIs, we were like, these aren't really database APIs. These are limited content delivery APIs, but we think that structured content should think like a database and work like a database, so we wanted those things. That was one of the reasons we couldn't use them.
Then we realized that, at least at that point, Contentful used Markdown for text. Then we saw loads of the people -- all two out in the field were using a different mold for text than other content. We were like, content is structured both in the kind of records but, basically, it's like one of the problems with Markdown. You have your YAML part and you have your Markdown part.
Simen: We think the text itself should function as structured content. We spent a lot of time making sure that we don't treat text any special way. You can actually curate. If you have a reference, let's say you have a text…. You use a reference in the text, so you actually don't just write a name. You reference. Let's say you reference an author in your blog and then, in the text, you will reference this person. Now, we can actually use Grok to query this document even though that content is actually in the running text. That was an important part for us. The text is not special. It's also structured.
Knut: I think, also, there are many front-end developers that listen to this podcast, right? Usually, when you get a Markdown blob. You have to use -- what's the React -- the dangerously … inter HTML or something.
Knut: Your rich text is always treated like this blog of stuff. Then you have to make container classes and stuff like that. When you have this portable text format that Simen was talking about, you can actually make separate like React components for different parts of that text. You can insert a class for that kind of paragraph or that kind of code block and stuff, stuff that, I guess, WordPress and Gutenberg are moving towards.
Simen: Yeah, it's like you have underline, perhaps, and you have a bold and italic emphasis. Then maybe you need to know whether a piece of text is an identifier. Let's say you're making a documentation site and you really want to be able to index this based on references to identifiers. You don't really want to know if it's supposed to look like code. It's not a style code you're looking for. You're looking for specifically an identifier. That's why we think it's really important with structured content that your text is also functioning as being able to power a kind of data driven accurate and also, like Knut said, then the really immediate whole thing is then you can create custom React components or whatever you're using in your front-end to capture that. They would never suddenly appear a bit of HTML.
A lot of our users in the beginning also came, of course, from being tasked with writing an app for some existing content and then realizing it was full of HTML, every now and then. Then, of course, getting rid of even that ability for editors to put in HTML, but still being able to invent these weird and interesting interaction things or visual things and being able to call that into your content. That was super important for us and was not possible with any of the existing services at the time.
Chris: What are you seeing people build with this? Surely, it looks like this Gatsby integration is pretty heavy, so I'm sure you see plenty of Gatsby sites and stuff. In a sense, you don't care, right? Front-end can do whatever it wants. You're just this structured data back-end. If front-end moves on from one technology to another, that's okay. Your data can come along with it, right?
Knut: Well, I guess we do care because we are also spending a lot of time making these tooling and stuff to make it easier to use the structured content that you get from the APIs. We made like the Gatsby source plugin. We are making libraries for rendering portable text to Markdown, BU, React, or whatever.
Simen: Then we have these superfans that want to use .NET and then write just fantastic C# adapters. If you really are motivated and want to use something that we are not familiar with or isn't on our radar yet, people do that and they will just fix it themselves.
It's exactly like you said. Philosophically, we don't care. From a marketing perspective, we care and try to make it easy for people to come from the popular languages and tools.
Simen: Also, it's a really powerful driver here is that companies are in-sourcing development. It used to be that you were like a big box sports retailer. You would go to a consultancy to get your website for your store whereas now, of course, they realize that that Web store is actually the important store. It's the retails like the high street retailers that are auxiliary.
Chris: So Next or Nuxt or anything?
Dave: Yeah. Yeah, or even, like, doesn't Phil Hawksworth from Netlify have some examples where he pulls down content and then generates a site? He pulls down a comment.json or something.
Chris: Yeah, it can be part of a build process, certainly, couldn't it? What do you actually see people doing, though? You surely can do that, right? You can just hit Sanity APIs during a build process. That seems fine, right?
Knut: Yeah, so I guess you see a lot of the same patterns as you see on the Web. People use Next like the isomorphic approach or Gatsby or … or something. But what's interesting is, also, we see these other use cases where people make scripts that renders to InDesign XML. Then they use Sanity with InDesign, which is mind-blowing to me.
More interestingly, when people have this structured content, they are like, "Oh, let's make a Slack bot that makes some of this content just available from a /command in Slack. We see a lot of this on the whim use, like, let's just make this little service that use some of the content to either push it in or take it out and stuff like that. That's super fun. Yeah. That's the fun part with having structured content, I think that you don't get with a Markdown file.
Dave: I hadn't even considered that. I could make a Dave bot for Slack. It's like, what does Dave think about X? It would just query my whatever.
Dave: My structured content and give me an answer.
Knut: Because working in a new document is just client.create and the object inside of it. Then you have the document available in the … so it's just as simple as that.
Dave: I mean it could be like your company wiki and now your helpdesk is a bot in your Slack. Where do I reset my email?
Simen: Yeah, it's funny. We had this company that wanted to use it as a kind of internal knowledge sharing tool. Then this guy had two weeks to make it. He was really stressed out when he was talking to me the whole time because this is a big company, so it was important to us. We needed to make them happy. Then he called me the day after. He was like, "I'm done."
Simen: Because … they were just used to CMS as the front-end.
Dave: I've got good news and bad news. The good news, I'm done. The bad news, I did not make much money. [Laughter] I finished way too early.
Knut: The interesting thing that comes up now is that we have this agency based in London. We had a meet up with them after JAMstack. They told us that they use Sanity for the whole design process. They integrated Sanity with Framer X. They had embedded the style guide in Sanity and then they used Framer X and all this stuff to prototype and move. That's what's kind of inspiring to see that these different ways of approaching design and digital development converge in a way.
Dave: That's always a problem in design, right? You're designing with optimistic data like Joe Smith is the name. It's not like Rama Chandra.
Simen: It's never 128 characters long and right to left. Yeah.
Dave: Yeah, and so now you have an actual database to vet. For me, it's always the post titles are too long.
Simen: With some of our early clients when we were making this, we actually had the content work start before. We created that with OMA and with MIT. We started the content work before we started the design work so that, when we started designing, we actually had everything, like a representative of everything on hand. Then it would become a back and forth process where they would look at the site and then maybe adapt the content. Maybe write longer or shorter text.
Knut: We used to do the same thing in … and it's the better way to work, I think, the content driven design.
Simen: I want to address one thing you talked about, like not making enough money because it's too simple, because that's interesting. I hear that sometimes, and I think it's a really wrong way to think because we, as consultants, the most annoying thing is when something is expensive because the budgets are usually the same either way. What's changing is how much value I can create with it. Then when I create more value, like with OMA, I really hit the nail with their website. They will come back and ask for the book and then I actually get more money because my value creation is so awesome.
This is, I think, part of why the JAMstack approach is so powerful because it's about radical convenience, making sure fewer people can do more. Then that's, in one sense as a consultancy, it looks like maybe you will make less money but, actually, more companies, more use cases, more situations will kind of then afford to make these kinds of tools, websites, or whatever and you will make more money, I think.
Dave: Very cool. I think we should wrap it up. Thank you so much, Simen and Knut. For those people who aren't following you and giving you money, how can they do that? Simen?
Simen: Well, yeah, yeah. I'm Svale, @Svale on Twitter. Then the rest of us, you can find in the footer at Sanity.io. You'll find our Slack and our Twitter and everything. Is that right?
Knut: That's right. Yeah. Go right to the footer. [Laughter]
Simen: Yeah, just go and there you'll find us.
Knut: Yeah, and I guess I have the worst Twitter handle.
Knut: Yeah, it's @kmelve. [Laughter] Go to the show notes. Yeah.
Simen: Really, join our Slack. That's a better way to follow us, right?
Knut: Yeah, I guess so.
Dave: Awesome. Thank you so much for coming on and giving us the debrief on a fully functioning headless CMS. That's pretty cool. Thank you, dear listener, of course, for downloading this in your podcatcher of choice. Be sure to star, heart, favorite it up. That's how people find out about the show. Follow us on Twitter, @ShopTalkShow, for tons of tweets a month. Share this show out. That'd be very, very helpful. We appreciate that. If you hate your job, head over to ShopTalkShow.com/jobs and get a brand new one because people want to hire people like you.
Simen: Hey, we need a full stack developer.
Dave: Chris, do you have anything else you'd like to say?
Chris: Oh, ShopTalkShow.com.