Our new slides: "Science of Culture? Computational analysis and visualization of patterns in cultural mage collections and datasets"

Science of culture? Computational analysis and visualization of cultural image collections and datasets from Lev Manovich

This PDF presents concepts, research questions and examples of computational analysis and visualizations of cultural image collections from our research lab (softwarestudies.com) created between 2009 and 2015. Visualized datasets include 20,000 images from MoMA photo collection, 773 Vincent van Gogh paintings, and 2.3 million Instagram images from 13 cities worldwide.

(Note that the original presentation has a few videos that are not part of this PDF document.)

"Instagram and Contemporary Image" - new book by Lev Manovich is released online

Lev Manovich, Instagram and Contemporary Image.

25,000 words. Written December, 2015 – January, 2016.

The complete book is released in parts on manovich.net (http://manovich.net/index.php/projects/tag:Article) during Winter-Spring 2016.

Text: Attribution-NonCommercial-NoDerivatives 4.0 International Creative Commons license. Images copyright belongs to their respective authors



Book chapters (when new parts are released, the links will be added here):


1) Subjects and Styles in Instagram Photography (part 1):

http://manovich.net/index.php/projects/subjects-and-styles-in-instagram-photography-part-1


2) Subjects and Styles in Instagram Photography (part 2):

http://manovich.net/index.php/projects/subjects-and-styles-in-instagram-photography-part-2



Subjects and Styles in Instagram Photography - abstract:

What are some of the types of Instagram photos today and how they relate to the 20th century photo culture? I analyze three common types of Instagram photos. We call these types "casual," "professional," and "designed." "Casual" photos are similar in function to personal photographers of the 20th century: they are created for friends; they privilege content of photos and ignore the aesthetics. Both “professional” and “designed” photo type are examples of what Alise Tifentale calls competitive photography. The difference is whom the authors compete with for likes and followers. The authors of professional photos aim for “good photo” aesthetics established in the second part of the 20th century, so they compete with other authors and lovers of such “classic” aesthetics including many commercial photographers. The authors of “designed” photos associate themselves with more “contemporary,” hip,” “cool” and “urban” lifestyle choices and corresponding aesthetics, so this is their peer group on Instagram.

The first part of Subjects and Styles in Instagram Photography chapter discusses the casual photo type. The second part discusses professional and designed photo types.

My text is an experiment to see how we can combine traditional “qualitative” approach of media theory and art history and newer quantitative analysis that uses “big cultural data” and computational methods. I draw on the analysis of 15 million images shared on Instagram in 16 global cities during 2012-2015 carried out in our lab (softwarestudies.com); results from other labs; my own informal observations from using Instagram for 3 years; and histories of photography, art and design.

Manifesto for Democratic Art History

Lev Manovich


Recently I was working on a series of articles about Instagram photography, and I wanted to find some digitized collection of personal 20th century photography to compare trends between then and now. I assumed that I could easily find at least 5000-1000 digitized images from some decades of the 20th century, and maybe even such samples for every decade. After searching for a while, I realized that nothing like this exists. The rare exhibitions and small collections of amateur photography all seem to reflect personal choices of particular collectors and curators. And they often only exhibit or collect certain amateur photos that look like art photography and modernist photography – in other words, the “usual.” The rest is discarded because it is too “usual.”

The same problem exists everywhere we look. What did hundreds of thousands of artists around the world painted in 1950? Or in 1850? What are the local differences around the world in personal photography in 1970? What were the contents and styles of all illustrations published in every magazine in 1930? Go ahead and ask a similar question about any form of visual culture, any medium, and any period until 2005. The answer will be the same: we have no idea. That is, we know (sometimes a lot) about the “leading” and “most important,” the “avant-garde” cases – which by definition are statistical outliers. (If they were forward looking avant-garde artists, this means that everybody else around them was different, so these artists do not represent general visual trends of the time.) Vincent van Gogh, Pablo Picasso, Berthe Morisot, Lyubov Popova, Diana Arbus, Ilya Kabakov, Ai Weiwei are all such outliers.

The disciplines that study visual culture are the “sciences of outliers.” But what about all other cultural artifacts?

But now situation is – or rather, could be – very different. We have social media. If we are interested in the histories of visual images, design, and photography - including the subjects, the styles, and the processes of creation and reception - social media is an unprecedented resource. In fact, a really broad, inclusive, and systematic study of visual culture becomes possible only after the wide adoption of social media and introduction of APIs by all major social networks (i.e. since 2005).

While modern histories of art, photography, visual design, and other visual media developed many useful concepts and theories, we can now see that they are mostly theoretical or descriptive undertakings that did not have sufficient data to study.

Imagine trying to develop discipline of biology based on only a few small habitats close to where you live, without being able to study the rest of the planet. And yet, this is what disciplines that study visual culture tried to do. They have developed their concepts and methods by studying particular small (and typically western) habitats. Moreover, within these selected habitats, they used very small and subjective samples. Their insights about these habitats based on the samples they used are often brilliant; but we don’t really know if any of this applies to any habitat as a whole and the rest of planet’s visual ecology.

Enter social media. Since 2005, anybody could easily and freely download massive numbers of images, descriptions, comments, location and time images were captured and shared, and other content and metadata from major social networks. (See “The Science of Culture? Social Computing, Digital Humanities, and Cultural Analytics.)

It is equally easy to create social media datasets that cover content shared in one city or in dozens of cities around the world. For example, for our 2013 project Phototrails, we downloaded 2.3 million Instagram photos shared in 13 global cities using a singe laptop. For 2014 project On Broadway we also used a single Mac to download 10.5 million images shared on Instagram over 5 months in New York City.

Because in a number of countries the percentages of social network users with different genders, ethnicities, incomes, educations and other demographic characteristics are close, this means that we can study creations and cultural activities of all such groups equally well. (For the latest data social media U.S., see Social Media Usage: 2005-2015. As of 2015, 68% of women and 62% of men in the U.S. used social media. The rates were 65% for whites, 65% for Hispanics and 56% for African-Americans. The rates for urban suburban and rural use were %64, %68, and %58. For the latest data on numbers of users of top global social networks, see Leading social networks worldwide as of January 2016.)

When we look back from this perspective of today, it becomes clear that almost all of cultural content created before 2005 is either has not been preserved, or, if it has been preserved, it is hard to access for research. The museums, the books, the institutional websites, and archives contain only tiny islands of all created images. Something only remains because somebody (typically elites) thought were important to preserve, or because somebody donated their collection, or because an institution got the grant to digitize, or because they now have market value. The later is often the most important reason.

Thus, Walter Benjamin’s words “history is written by the victors” certainly perfectly apply to history of images. While works of a handful of “great” artists and photographers are studied, exhibited, and widely published, almost everything else has become invisible. To use a different methodology, we can say that only “top 40” are showcased, and “long tail” of visual culture has been cut off.

Even if you devote a few years to visit all collections around the world relevant to your topic – which requires an institutional affiliation and funding - there is no guarantee that they contain a balanced sample of images for this topic. And while some parts of these islands have been now digitized and available via institutional websites, such websites quickly age, the technology changes and after a while they no longer work.

This digitization process so far did not address the key problem of our “historical storage” – that the material related to a given topic, period, style, subject and so on exists across many separate physical archives, making it next to impossible to see everything. (Important large scale projects such as europeana.eu are working on this problem, but it is not clear if it will be ever be solved).

But most importantly, the distribution of preserved history (and consequently, the digitized materials) is very uneven geographically, with richer contrives having preserved (and digitized) more of their visual artifacts than more poor countries. And in terms of what has been preserved, many of our museum collections consists from the objects that came from the houses of the upper classes (typically, aristocracy) as opposed to lower classes. What world museums present today to millions of visitors as “history of art” are the commissions and possessions of the kings and tsars, not the creations of the people. (Similarly, global art world revolves around the taste of only a few thousand collectors who compete among themselves in purchasing artworks for six figure prices.)

Certainly, social media has its own significant limitations. It is not %100 inclusive.
The rates of use vary between geographic areas around the world, and between age groups. Not every kind of images ends up being shared in social networks. Some local visual communities and professional groups tare not using global social networks. Not every network has easy to use API. (However, since we can also scrape content of numerous websites, this gives us additional resources.)

But in comparison to what we preserved from before 2005, and also in terms of ease of access and quality of records, social media is God’s gift to researchers. It allows us for the first time to study visual culture created by many groups in society. In many countries, we can now study images created by %30-%60 of the whole population of a given country - as opposed to only tiny numbers of professional creators privileged by market, cultural elites, or particular cultural values that control what is saved and what is deleted.

People like to talk about how social networks “exclude” many from participating. But if we can study images shared by %30-%60 of people in a given country, this to me looks like a real progress in comparison to what academic disciplines and cultural institutions have studied and exhibited until now: typically only a few hundred artists, filmmakers, photographers per country (or even less).

Usually these were culture professionals who lived in majorly cities and this allowed them to become successful. But social media allows us to study what people create regardless of where they live. And it does not matter if a person is followed by 50,000 other users or by 5 - their content is equally accessible via APIs.

Only now we can start seeing both broad patterns manifested across billions of images, and also numerous smaller patterns that may manifest themselves only across a few hundred thousands. Only now we can start seeing how preferences in content, styles, and cultural sensibilities change gradually over short periods of times. Only now we can study numerous visual subcultures across the world. Only now we can study reliably the patterns of borrowing and circulation, copying and mutation, sameness and variability. Only now we can pose hypotheses and then verify or reject them, as opposed to promoting our intuitions as expert knowledge.

In short, only now art history, photo history, design history and all other image histories become truly possible.


REFERENCES:

The Evolution of the Social Media API,” http://mashable.com/2009/05/21/social-media-api/

"The Science of Culture? Social Computing, Digital Humanities, and Cultural Analytics, http://manovich.net/index.php/projects/cultural-analytics-social-computing, 2015.

Phototrails, http://phototrails.net, 2013.

On Broadway, http://on-broadway.nyc, 2014.

"Social Media Usage: 2005-2015," http://www.pewinternet.org/2015/10/08/social-networking-usage-2005-2015/, 2015.

"Leading social networks worldwide as of January 2016, ranked by number of active users (in millions)," http://www.statista.com/statistics/272014/global-social-networks-ranked-by-number-of-users/, 2016.

Europeana Collections, http://europeana.eu/

a new article by Alise Tifentale and Lev Manovich: "Competitive Photography and the Presentation of the Self"

PDF:
Alise Tifentale and Lev Manovich, Competitive Photography and the Presentation of the Self

Publication information:
"#SELFIE–Imag(in)ing the Self in Digital Media," edited by Jens Ruchatz, Sabine Wirth, and Julia Eckel. Marburg, 2016 (forthcoming).


Abstract:

Many discussions of photography and other types of visual culture including user-generated content often rely on professional — amateur distinction. In this article we introduce a different pair of concepts: competitive — non-competitive. We believe that analyzing photography history and its present such as Instagram’s visual universe using these new concepts allows us to notice phenomena and patterns that traditional professional—amateur distinction hides.

The analysis of presentation of self in online digital photography is a case in point. We can now see that the selfie genre is complemented by an “anti-selfie” genre that presents the self in a different way. The two genres correspond to different understanding and uses of Instagram by non-competitive and competitive photographers.


Information about the illustration above:
Examples of Instagram “anti-selfies.” Authors, first row: @vita_century, @vita_century, @deroy_night. Second row: @lavimeer, @merciless_mart, @sex_on_water. Third row: @recklesstonight. Author’ age ranges from 14 to early 20s. (as explicitly indicated in their profile or can be estimated from photos in their galleries). The authors live in Russia or Ukraine. The photos were shared during Summer and Fall 2015.

Lev Manovich's new lecture slides: "The Science of Culture? Theory and examples of computational cultural analysis using big image data"

This the latest version of my 2015 lecture about some theoretical ideas and our projects visualizing big image data (November 15):

Keynote file, 212 Mb.

PDF file, 90 MB (without video)


Here is the slides view as a single image (89 slides):

MoMA, New York Public Library, Google, and Twitter: our 2014-2015 projects, exhibitions, publications

The last 24 months have been a very busy and productive time for our lab.

In 2014, we created six new projects focusing on analysis, visualization, and interpretation of large cultural data. Four of them were installations commissioned by New York Public Library, Google Zeitgest 2014 conference, National Taiwan Museum of Fine Arts, and a media festival in Sao Paulo. We were also commissioned to create a visualization for Wired August 2014 issue.

In 2015, we are showing our projects in 12 exhibitions in the U.S., Europe, and Asia, including Infosphere in ZKM (Center for Art and Media), Tallinn Architecture Biennale, Data Traces in Riga, and West Bund Biennial of Architecture and Contemporary Art in Shanghai. We also created a new selficity London edition for Big Bang Data exhibition at Somerset House in London.

Our projects were featured in hundreds of publications including New York Times, Wall Street Journal, BBC, The Guardian, CNN, PBS, NBC, LA Time, Washington Post, San Francisco Chronicle, The Atlantic, Discovery Channel, National Geographic, Wired, Slate, Fast Company Co.Design, Spiegel, Le Monde, Buzzfeed, Gizmodo. For the full list of media coverage, see http://manovich.net/index.php/press

We are grateful to institutions that are supporting our work: The Graduate Center at City University of New York; California Institute for Telecommunication and Information and The Andrew Mellon Foundation.

We were lucky to work with the amazing team of data and media designers: Moritz Stefaner, Daniel Goddemeyer, and Dominikus Baur. Moritz is the creative director and visualization designer for both Selfiecity and On Broadway, and we are very grateful for all his work and dedication to these projects.

Awards and recognition:

Selfiecity received Gold (in website category) from Information Is Beautiful 2014 Awards.

Lev Manovich is included in The Verge 50: the list of the most interesting people building the future (2014).

Selfiecity is included in six best visualizations of 2014 lists.

Twitter Data Grant:

Out of 1300 international teams who applied, only six teams received the Twitter Data Grant. ">Twitter Data grants. As the part of the grant, Twitter gave us access to every tweeted image worldwide for 2011-2014. We will be publishing the results of our analysis of this amazing dataset in 2015-2016.

Culture Analytics Institute

Lev Manovich is a member of organizing team for Culture Analytics Institute (March 6 - June 10, UCLA). 115 leading academic and industry researchers who will be speaking at 4 Institute workshops.

Commissioned projects:

On Broadway

An interactive installation that visualizes multiple layers of data and images collected along the 13 miles of Broadway spanning Manhattan. Data includes 660,000 Instagram images, Twitter posts and images, Foursquare check-ins, Google Street View images, 22 million taxi pickups and drop-offs (2013), and economic indicators from the U.S. Census Bureau. Artists: Daniel Goddemeyer, Moritz Stefaner, Dominikus Baur, Lev Manovich. The installation is on view at the New York Public Library as part of the exhibition Public Eye: 175 Years of Sharing Photography, December 12, 2014 - January 3, 2016. Interactive web app is available at On Broadwaywebsite.

Phototrails: Animated

Animated visualizations of 2.3 million Instagram images from 13 global cities commissioned for Google Zeitgest 2014 Conference. Artists: Nadav Hochman, Lev Manovich, Jay Chow. September 14-16, Paradise Valley, Arizona. Original project (2013): Phototrails.

SelfieSaoPaulo

A site-specific project for a media facade in Sao Paulo, Brazil. Instagram selfies from Sao Paulo are animated to reveal patterns in self-representation. One animation presents the photos organized by estimated age, another by gender, and the third by the degree of smile. Artists: Jay Chow, Lev Manovich, and Moritz Stefaner. SelfieSaoPaulo was on view on the media facade, located at the building FIESP / SESI and Alameda das Flores (Avenida Paulista 1313), every night for the duration of the 2014 SP_Urban Festival in Sao Paulo, Brazil. June 9 – July 7, 2014.

Taipei Phototime

Real time visualization of Instagram image streams Taipei and New York. Artists: Lev Manovich and Jay Chow. Taipei Phototime was commissioned for Wonder of Fantasy. 2014 International Techno Art Exhibition at the National Taiwan Museum of Fine Arts, Taipei. May 17 – August 3, 2014.

selfiecity London

Analysis and visualization Instagram selfies from London. Commissioned for Big Bang Data exhibition at Somerset House in London. December 3 2015 – February 28, 2016.

Self-initiated projects:

The Exceptional and the Everyday: 144 hours in Kyiv

The first project to analyze the use of Instagram during a social upheaval. Using computational and data visualization techniques, we explored 13,208 Instagram images shared by 6,165 people in the central area of Kyiv during 2014 Ukrainian revolution (February 17 - 22, 2014). Team: Lev Manovich, Jay Chow, Alise Tifentale, and Mehrdad Yazdani, with a guest essay by Elizabeth Losh. Released 10/07/2014.

Selfiecity

Analysis and visualization of 3,200 Instagram selfies from five global cities. The project includes an interactive web app Selfiexploratory. Team: Lev Manovich, Dominikus Baur, Jay Chow, Daniel Goddemeyer, Nadav Hochman, Moritz Stefaner, Alise Tifentale, and Mehrdad Yazdani. Released 02/19/2014.

Publications: new books 2014-2015:

Data Drift: Archiving Media and Data Art in the 21st Century. Editors: Rasa Smite, Raitis Smits, Lev Manovich. (RIXC, 2015)

The Illusions. Digital Original Edition. A BIT of The Language of New Media. Cambridge, Mass.: MIT Press, 2014.

Publications: articles, book chapters, conference papers 2014-2015:

Manovich, Lev. The Science of Culture? Social Computing, Digital Humanities, and Cultural Analytics in The Datafied Society. Social Research in the Age of Big Data, edited by Mirko Tobias Schaefer and Karin van Es. Amsterdam University Press, forthcoming in 2016.

Manovich, Lev. Exploring Urban Social Media: Selfiecity and On Broadway in Code and the City, edited by Robert Kitchin and Sung-Yueh Perng. Routledge, forthcoming February 2016.

Manovich, Lev, and Mehrdad Yazdani. Predicting Social Trends from Non-photographic Images on Twitter, Big Humanities Data workshop at IEEE Big Data 2015. Forthcoming in Proceedings of 2015 IEEE International Conference on Big Data.

Heftberger, Adelheid. «Die Verschmelzung von Wissenschaft und Filmchronik». Das Potenzial der reduktionslosen Visualisierung am Beispiel von «Das elfte Jahr» und «Der Mann mit der Kamera» von Dziga Vertov. In La visualisation des données en histoire. Visualisierung von Daten in der Geschichtswissenschaft , edited by Enrico Natale, Christiane Sibille, Nicolas Chachereau, Patrick Kammerer, and Manuel Hiestand. Zürich: Chronos-Verlag, 2015.

Manovich, Lev. Data Science and Computational Art History. International Journal for Digital Art History, Issue 1, 2015, pp. 12-34. Featured article.

Manovich, Lev and Everardo Reyes. “Info-aesthetics,” in 100 Notions for Digital Art, edited by M. Veyrat. Paris: Les Editions de l’Immatériel.

Manovich, Lev. The Language of Media Software. In The Imaginary App, edited by Svitlana Matviyenko and Paul D. Miller (Cambridge, Mass.: The MIT Press, 2014), pp. 189-204.

Manovich, Lev. Software is The Message. Journal of Visual Culture, Volume 13, Issue 1, pp. 79-81.

Reyes, Everardo. Aesthetics of temporal and spatial transformations in environments, Metaverse Creativity, Volume 4, Issue 2, pp. 151-165.

Reyes, Everardo. Explorations in Media Visualization (invited talk) in Extended Proceedings of the 25th ACM conference on Hypertext and Hypermedia Hypertext'14, September 2014. New York: ACM Press.

da Silva, Cicero Inacio and Marcio Santos. Analysing big cultural data patterns in 2.200 covers of Veja Magazine. Proceedings of the Digital Humanities Congress 2012. University of Sheffield, 2014.

Hochman, Nadav and Lev Manovich. A View From Above: Exploratory Visualizations of the Thomas Walther Collection, in Object:Photo. Modern Photographs 1909–1949: The Thomas Walther Collection. Museum of Modern Art (MOMA), website.

Manovich, Lev, Jay Chow, Alise Tifentale, Mehrdad Yazdani. The Exceptional and the Everyday: 144 Hours in Kyiv, in Proceedings of 2014 IEEE International Conference on Big Data, 77-84.

Tifentale, Alise and Lev Manovich. Selfiecity: Exploring Photography and Self-Fashioning in Social Media, in Berry, David M. and Michael Dieter, eds, Postdigital Aesthetics: Art, Computation and Design (Palgrave Macmillan: 2015), pp. 109-122.

Hochman, Nadav. The social media image, Big Data and Society, July-December 2014.

Hochman, Nadav, Lev Manovich, and Mehrdad Yazdani. On Hyper-Locality: Performances of Place in Social Media, presented at The International AAAI Conference on Weblogs and Social Media (ICWSM 2014).

Manovich, Lev. Watching the World, Aperture, No. 214 (2014), special issue on “Documentary, Expanded.”

new book "Data Drift: Archiving Media and Data Art in the 21st Century" (editors: Rasa Smite, Raitis Smits, Lev Manovich)

Data Drift: Archiving Media and Data Art in the 21st Century

Editors: Rasa Smite, Raitis Smits, Lev Manovich
Publisher: RIXC, LiepU MPLab (October 8, 2015)
Language: English
ISBN-10: 9934843439
ISBN-13: 978-9934843433
Paperback: 296 pages
Product Dimensions: 8.4 x 5.8 x 0.7 inches
Available on Amazon



Malraux's imaginary museum has become a reality of our digital age. Everyday, a massive amount of new visual, textual, and transactional data is added to this vast archive. How do we decide what bits of contemporary digital culture to preserve, study and curate given that its universe is constantly expanding?

The parallel questions of selection, archiving and curating are important to address with regards to all recent media art forms - from early video, computer and telematic art to later networked, web-based, software, and data art. There are several strategies already used in media art archiving practice today. However, it is also important to look for even more new approaches. For instance, data visualization opens up new possibilities for archiving, reinterpreting and exhibiting artworks, allowing us to look at the past and present from other unfamiliar perspectives.

The full color 296 page book combines a selection of papers presented in the Media Art Histories: Renew conference, selections from Save As exhibition, and new texts written for this volume. It also contains the catalog of Data Drift exhibition of data art and data design (curated by Rasa Smite, Raitis Smits, Lev Manovich). The book is published in connection with the RIXC annual 2015 festival that took place in Riga, Latvia.


Contributors:

Frieder Nake
Rasa Smite
Raitis Smits
Lev Manovich
Pau Alsina
Clarisse Bardiot
Valentino Catricala
Michael Century
Hanna Barbara Hoelling
Francesca Franco
Esteban Garcia
Rainer Groh
Tjarda de Haan
Franziska Hannss
Chris Hales
Canan Hastik
Vanina Hofman
Nils Jean
Rudi Knoops
Paul Landon
Esther Lapczyna
Laura Leuzzi
Magdalena Anna Nowak
Ianina Prudenko
Georgina Ruff
Arnd Steinmetz
Bernhard Thull
Elio Ugenti
Paul Vogel
Joanna Walewska
Reba Wesdorp
Artemis Willis
Erkki Huhtamo

Our new paper "Predicting Social Trends from Non-photographic Images on Twitter" accepted for IEEE 2015 Big Data Conference

Random sample of Twitter images from 2013 labeled by GoogLeNet deep learning model as web sites and texts. We call refer to these images as "image-texts." The category includes screen shots of text chats, other types of texts and other kinds of non-photographic images. We found that the frequencies of these images are correlated with well-being responses from Gallup surveys, and also median housing prices, incomes, and education levels.


Johannes Vermeer. Woman in Blue Reading a Letter. 1663-1664.



Our new paper has been accepted for Big Data and the Humanities workshop at IEEE 2015 Big Data Conference:



Predicting Social Trends from Non-photographic Images on Twitter

Mehrdad Yazdani (California Institute for Telecommunication and Information) and Lev Manovich (The Graduate Center, City University of New York)



Abstract

Humanists use historical images as sources of information about social norms, behavior, fashion, and other details of particular cultures, places and periods. Dutch Golden Era paintings, works by French Impressionists, and 20th century street photography are just three examples of such images. Normally such visuals directly show objects of interests such as social scenes, city streets, or peoples dresses. But what if masses of images shared on social networks contain information about social trends even if these images do not directly represent objects of interest? This is the question we investigate in our study.

In the last few years researchers have shown that aggregated characteristics of large volumes of social media are correlated with many socio-economic characteristics and can also predict a range of social trends. The examples include flu trends, success of movies, and measures of social well-being of populations. Nearly all such studies focus on text content, such as posts on Twitter and Facebook.

In contrast, we focus on images. We investigate if features extracted from Tweeted images can predict a number of socio-economic characteristics. Our dataset is one million images shared on Twitter during one year in 20 different U.S. cities. We classify the content of these images using the state-of-the-art Convolutional Neural Network GoogLeNet and then select the largest category that we call “image-texts” - non-photographic images that are typically screen shots of websites or text-message conversations. We construct two features describing patterns in image-texts: aggregated sharing rate per year per city, and the sharing rate per hour over a 24-hour period aggregated over one year in each city.

We find that these features are correlated with self-reported social well-being responses from Gallup surveys, and also median housing prices, incomes, and education levels. These results suggest that particular types of social media images can be used to predict social characteristics not readily detectable in images.


The full paper will be available online after IEEE 2015 Big Data conference.

New essay about Maidan revolution and Instagram is available from "144 hours in Kyiv" project

A selection from all Instagram images publically shared in the central part of Kiev in the evening of February 18, 2014.


Last Fall we published a research project The Exceptional and the Everyday: 144 hours in Kyiv. This was the first project to analyze use of Instagram during a social upheaval. We used computational and visualization techniques to analyze over 13,000 Instagram images shared by people in Kyiv during one week of the Euromaidan Revolution in February 2014.

This revolution took place exactly one year ago - but it is not over yet. Immediately after the success of revolution events in February 2014, Russia sent its troops to annex Crimea, and later to support separatists in Eastern Ukraine. Russian military intervention in Ukraine continues, despite all international efforts to stop it.


Today we are publishing a new essay on our 144 hours in Kyiv project site:

Liquid Categories for Augmented Revolutions

Dr. Svitlana Matviyenko, University of Western Ontario

The essay is written by Dr. Svitlana Matviyenko. Born in Ukraine, she became the most interesting young literary critic in Kyiv. Currently she is completing her second PhD and teaching at the University of Western Ontario (Canada). While we worked on the project last year, we asked Svitlana to help us understand the use of Ukrainian tags in our Instagram dataset, given her connections in Kyiv and understanding of Maidan movements. Her insights were so interesting that we asked her to develop a longer text. The essay she wrote is very interesting and original - you must read it if you are interested in social movements, use of social media, contemporary media theory or events in Ukraine. We produced a few new data visualizations specifically for Svitlana, and I am happy to see that she put them to good use in her essay.

The Imaginary App: a new book from Software Studies series @ The MIT Press

The latest book from Software Studies series at The MIT Press:


The Imaginary App

Edited by Paul D. Miller (aka DJ Spooky that Subliminal Kid) and Svitlana Matviyenko. The MIT Press, 2014.


From the publisher:

Mobile apps promise to deliver (h)appiness to our devices at the touch of a finger or two. Apps offer gratifyingly immediate access to connection and entertainment. The array of apps downloadable from the app store may come from the cloud, but they attach themselves firmly to our individual movement from location to location on earth. In The Imaginary App, writers, theorists, and artists--including Stephen Wolfram (in conversation with Paul Miller) and Lev Manovich--explore the cultural and technological shifts that have accompanied the emergence of the mobile app. These contributors and interviewees see apps variously as “a machine of transcendence,” “a hulking wound in our nervous system,” or “a promise of new possibilities.” They ask whether the app is an object or a relation, and if it could be a “metamedium” that supersedes all other artistic media. They consider the control and power exercised by software architecture; the app’s prosthetic ability to enhance certain human capacities, in reality or in imagination; the app economy, and the divergent possibilities it offers of making a living or making a fortune; and the app as medium and remediator of reality.

Also included (and documented in color) are selected projects by artists asked to design truly imaginary apps, “icons of the impossible.” These include a female sexual arousal graph using Doppler images; “The Ultimate App,” which accepts a payment and then closes, without providing information or functionality; and “iLuck,” which uses GPS technology and four-leaf-clover icons to mark places where luck might be found.


Contributors:

Christian Ulrik Andersen, Thierry Bardini, Nandita Biswas Mellamphy, Benjamin H. Bratton, Drew S. Burk, Patricia Ticineto Clough, Robbie Cormier, Dock Currie, Dal Yong Jin, Nick Dyer-Witheford, Ryan and Hays Holladay, Atle Mikkola Kjøsen, Eric Kluitenberg, Lev Manovich, Vincent Manzerolle, Svitlana Matviyenko, Dan Mellamphy, Paul D. Miller aka DJ Spooky That Subliminal Kid, Steven Millward, Anna Munster, Søren Bro Pold, Chris Richards, Scott Snibbe, Nick Srnicek, Stephen Wolfram.


About Software Studies series at MIT Press:

The Software Studies series publishes the best new work in a critical and experimental field that is at once culturally and technically literate, reflecting the reality of today’s software culture. The field of software studies engages and contributes to the research of computer scientists, the work of software designers and engineers, and the creations of software artists. Software studies tracks how software is substantially integrated into the processes of contemporary culture and society. It does this both in the scholarly modes of the humanities and social sciences and in the software creation/research modes of computer science, the arts, and design.


Software Studies series co-editors:

Dr. Noah Wardrip-Fruin, The University of California, Santa Cruz (UCSC).

Dr. Lev Manovich, The Graduate Center, City University of New York (CUNY).




The cover of The Imaginary App

"The Social Media Image" - a new paper from Nadav Hochman (Phototrails project)

Imageplot of 50,000 Instagram images from Paris (spring 2013) organized by average brightness and contrast.


The new paper by Nadav Hochman (see Phototrails project) has been published in BIG DATA and SOCIETY:

The Social Media Image

Download full paper text

Abstract:

How do the organization and presentation of large-scale social media
image sets in social networking apps affect the creation of visual
knowledge, value, and meaning?

The article analyzes fundamental elements in the changing syntax of
existing visual software ontology — the ways current social media
platforms and aggregators present and categorize social media images.
It shows how visual media created within social media platforms
follow distinct modes of knowledge production and acquisition.

First, I analyze the structure of social media images within data streams as
opposed to previous information organization in structured databases.
While the database has no pre-defined notions of time and thus
challenges traditional linear forms, the data stream re-emphasizes the
linearity of a particular data sequence and activates a set of new
relations to contemporary temporalities.

Next, I show how these visual arrangements and temporal principles are
manifested and discussed in three artworks: Untitled (Perfect
Lovers) by Felix Gonzalez-Torres (1991), The Clock by
Christian Marclay (2011), and Last Clock by Jussi Ängeslevä and
Ross Cooper (2002).

By emphasizing the technical and poetic ways in which social media
situate the present as a “thick” historical unit that embodies
multiple and synchronous temporalities, the article illuminates some
of the conditions, challenges, and tensions between former visual
structures and current ones, and unfolds the cultural significations
of contemporary big visual data.

A Window, a Message, or a Medium? Learning from Instagram

My upcoming talk at International Conference on Mobile and Social Media Practices (organized by Dr. Tristan Thielmann).

June 19-21, 2014
The University of Siegen
Germany


Title:

A Window, a Message, or a Medium? Learning from Instagram


Abstract:

Over last few years, tens of thousands of researchers in social computing and computational social sciences started to use available data from social networks and media sharing services (such as Twitter, Foursquare and Instagram) created by users of mobile platforms. The research uses techniques from statistics, machine learning, and visualization, among others, to analyze all kinds of patterns contained in this data and also (less frequently) propose new models for understanding the social. The examples include analysis of information propagation in Twitter, predicting popularity of photos on Flickr, proposing new sets of city neighborhoods using Foursquare users check-ins, and understanding connections between musical genres using listening data from Echonest.

In my talk I will address a fundamental question we face in doing this research: what exactly are we learning when analyzing massive social media data? Is it a window into real-world social and cultural behaviors, a reflection of lifestyles of particular demographics who use mobile platforms and particular network services, or only an artifact of mobile apps? In other words - is social media a "message" or a "medium"?

I will discuss this question using three recent projects from my lab (softwarestudies.com). The projects use large sets of Instagram images and accompanying data together with data science and visualization tools. Phototrails.net (2013) analyzes 2.3 million photos from 13 global cities to investigate how different kinds of events are represented in these photos. The project also investigates if the universal affordances of Instagram app (same interface and same set of filters available to all users) result in universal digital visual language. Selfiecity.net (2014) analyzes the distinct artifact of mobile platforms – selfies. We compare thousands of selfies to see if cultural specificity of different places and cultural is preserved in this genre. Finally, our third project compares Instagram photos taken by visitors in a few major modern art museums, asking if photographs of famous works of the art differ depending on what these artworks are and where they are situated.

Gender, age, and ambiguity of selfies on Instagram

selfiecity.net research update by Mehrdad Yazdani, Research Scientist, Software Studies Initiative.



Who are the people behind selfies? Are they mostly young? Do women prefer taking selfies over men? Do these variations depend on geographic location? We looked at over 4,500 selfies from six cities to gain a sense of the different age groups and genders. (Analysis and visualizations of our findings for 3200 images from five cities from this dataset are available on selfiecity.net.)

We did this by first downloading a random sample of 140,000 images among all Instagram photos shared by people in central areas of 6 global cities for one whole week (Dec 4-12, 2013). Our random sample of Instagram photographs include:

• 30,000 images from Tokyo
  
• 30,000 images from New York
  
• 20,000 images from Bangkok
  
• 20,000 images from Berlin
  
• 20,000 images from Moscow
  
• 20,000 images from Sao Paulo

Now we have to figure out which of these images are actual selfies. We define the selfie as a photograph that you take of yourself. Since this definition can lead to a great deal of ambiguity, we ask several people to gain a better consensus. We utilize Amazon's Mechanical Turk service to find human reviewers (who are payed!) to review each image. At least 3 reviewers review each of the 140,000 images to find all the selfies. Obviously, in some cases, the reviewers disagree if an image is a genuine selfie or not. To resolve these disagreements, we use a simple majority vote (the mode of votes in statistics jargon) to make the final call as to whether the image is a selfie or not.

What you see above is what we call the "selfie rate," that is, the percentage of selfies that our reviewers from Mechanical Turk found from the 140,000 images that we collected. What is most striking about this figure is that, in contrast to popular belief, the selfie is not ubiquitously plastered all over Instagram. In fact, Sao Paulo has a selfie rate clocking in at just under 5%! Tokyo, on the other hand, has an even significantly lower selfie rate of a hair above 1%.

But we don't stop at just finding selfies from our set of images. If a reviewer thinks that the image is indeed a selfie, he or she also takes a best guess at the gender and age of the selfie. Again, these are reviewers who use Mechanical Turk on a regular basis and therefore asking them to complete an image-tagging problem is on-par with their expertise. The graph above shows that the gender distribution of the selfies is heavily skewed towards females. Moscow in particular has a large disproportionate amount of female selfies. In fact, it is 4 times less likely that a selfie from Moscow is male (with a 95% confidence interval between 3.3 and 5.3).

However, it is not fair to assume that gender is a binary factor that we can neatly divide into "male" or "female." Would it be possible for us to have a way of measuring the ambiguity of a selfie's gender? Answering such a question is extremely difficult, but let's take a data science approach (read: hack). We will make an assumption that if it is difficult to ascertain a selfie's gender as "male" or "female" then our reviewers from Mechanical Turk will have a harder time making a decision. Since we have multiple reviewers (at least 3 or more), then there will be more disagreements if it is truly difficult for the reviewers to determine the selfie's gender. Let's assign a confidence score between 0 and 1 to the collective agreement of the reviewers for the gender of the selfie. What follows are the averages of gender discrimination confidence for the different cities:

We see some very interesting patterns emerging from this figure. Over the entire population, we see that the reviewers are fairly confident (over 95%) of a selfie's gender. However, consistently for every city, the average gender confidence for males is less than those of females. In the case of Berlin, this difference may very well be insignificant and due to chance, but for the other cities we see much wider gap in confidence. Especially in the case of Sao Paulo and Moscow, the reviewers are much more confident at detecting females than the other cities. One possible interpretation: What makes these cities unique is that women in these cities are unquestionably "female looking" (at least when they take their selfies and post them), so the confidence reviewers have for these female selfies is higher.

We next take a look at the age distributions of the selfies. Here they are organized by city and gender:

The most dramatic result here is that for every city we see that men who take selfies are older than their female counterparts. Bangkok has the youngest selfie enthusiasts, while New Yorkers have the oldest. If we look on a log-scale, as the age of a selfie increases, the odds of the selfie being male increases by a factor of 6.7 (with a 95% confidence interval between 4.99 and 9.03). Overall, however, the early twenty somethings dominate selfies on Instagram. As before, we determine the age of the selfie by asking several reviewers to make their best guess. We then estimate the age of the selfie by taking the median of the guesses of the reviewers. As in the case for determining gender, this can be a very difficult task and certain selfies can be harder to answer. To ascertain the agreement level for a selfies age, we computed the standard deviation of the reviewers guesses. In this case, higher standard deviation suggests more disagreement among the reviewers. We refer to this standard deviation as the "disagreement." Below we show average disagreements for each city and gender:

With the exception of Berlin and New York (that have the highest disagreements), female age discrimination has the least amount of disagreement. The difference between the disagreement levels of males and females in Berlin does not appear to be significant. By far, Bangkok has the least amount of disagreement for age discrimination of female selfies among all cities. It is difficult to ascertain why this is the case. We welcome any hypotheses for this finding!

In summary, our study suggests that selfies are not the dominant imagery shared on Instagram. We have also observed that the selfies are extremely popular by females and twenty somethings. We are planning more posts on softwarestudies.com about additional details and more results from selfiecity.net research, so check back to see them.

An Outline for Computational Art History

The Hawaiian Star, 5930 front pages, 1893-1912 (Vimeo). Created by UCSD undergraduate student Cyrus Kiani in Manovich's visualization class, 2012.



An Outline for Computational Art History


Computational art history - use of algorithms for the analysis and visualization of patterns in art production, dissemination, reception/interaction, and scholarship. (In other words: use of computers to augment human intellect and intuition.)


Three complementary ways to think about research projects and methods in computational art history:


1 | Which stage(s) in art circulation process do you want to analyze?
- production [example 1, example 2]
- dissemination [example]
- reception (visitors movements in a museum, user tags, professional art criticism publication, user searches, etc.) [example]
- user interaction with digital media [many possibilities for capturing data about the experience; the work is co-created by software and participant]
- institutions / exhibitions / collecting / publications / art sales / art world system [example 1, example 2]
- scholarship

and also:
influence [example]
artists networks [example]
interactions with other cultural areas
etc.


2 | What data types you want to analyze?
structured, unstructured
text, images, video, 3D shapes, spaces, networks of relations, geospatial data, movements capture, interactions capture


3 | Which stage(s) in data analysis workflow you will work on?
- acquire and clean data (media agnostic)
- organize data for analysis (media agnostic)
- feature extraction (automatic) (media specific) / adding metadata (media agnostic)
- exploratory visualization (media agnostic)
[however images and video allow for unique visualization techniques: example 1, example 2]
- (optional) data analysis using classical statistics and/or data science methods (media agnostic)





Lev Manovich, february 11, 2014.

"Software, Globalization and Political Action" course co-taught by Manovich and Buck-Morss, Spring 2014, Graduate Center CUNY

#softclassgc (class Twitter hashtag)


SYLLABUS (Google Doc updated throughout the semester - we suggest you check it every weekend to see the updates)


Readings (Dropbox)







Top: a frame from A Man with a Movie Camera by Dziga Vertov, 1929.
Bottom: Visualization of locations of large number of photos in NYC shared on Flickr, from "Locals and Tourists" by Erik Fisher, 2009).


Spring 2013 course: Software, Globalization, and Political Action

Co-taught by Susan Buck-Morss (Political Science) and Lev Manovich (Computer Science). The Graduate Center, City University of New York (CUNY)

Tuesdays 2-4pm.
4 Credits. Cross-listed in the Programs in Political Science, Computer Science and Art History.



Description:

This interdisciplinary seminar will explore concepts and methods from both critical theory and software studies. It is taught by Prof. Susan Buck-Morss (Political Science), and Prof. Lev Manovich (Computer Science).

We will cover three themes:

1) Vision and Image - From Walter Benjamin and Dziga Vertov to computer vision, Google Earth, Adobe Creative Suite, and Instagram: new strategies of seeing and representation in modern and software societies. Image v. Concept (Hegel against ‘picture thinking’) Image and historical matter (Benjamin on the “dialectical image”). Aesthetics and Politics: Images as a (trans-local) language for political action; vision and democracy: the “ethical turn.” Digital Image Processing and Computer Vision and examples of their application in creative industries, vernacular digital photography, and digital humanities.

2) Data and Knowledge - Knowledge production in the age of "big data." Images as sources of knowledge. Political critique of methods (positivism, abstraction, categorical givens) and goals (surveillance, marketing, positivism). Knowledge of, by and for whom? Data science as the new key technology for production of knowledge and decision making in big data societies. Data visualization as a research method in humanities and social sciences (including art history and political science). From representing reality to representing data. Data art.

3) Crowds and Networks - What are the new forms of sociality and political action enabled by global networks? Networked Images as political instruments. Crowds and the de-centered brain. Crowds and/as a medium of global political action since the Arab Spring. The new body politic as a body without skin. Social networks and computational social science. Social media analytics. Artistic visualizations of social data.

Looking back: 2013 publications and projects from Software Studies Initiative

Projects:

Phototrails: Analysis and visualization of 2.3 million Instagram photos shared by people in 13 global cities. Released 7/1/2013. Team: Nadav Hochman, Lev Manovich, Jay Chow.


Exhibitions:

The Aggregate Eye: 13 cities / 312,694 people / 2,353,017 photos. Amelie A. Wallace Gallery, October 29 – December 5, 2013. Artists: Nadav Hochman, Lev Manovich, Jay Chow. Curators: Hyewon Yi and Alise Tifentale.


Books:

Manovich, Lev. Software Takes Command. Bloomsbury Academic, published 7/4/2013. 100,000 words.


Articles and book chapters (13):


Akdag Salah, A.A., L. Manovich, A.A. Salah and J. Chow, "Combining Cultural Analytics and Networks Analysis: Studying a Social Network Site with User-Generated Content." Journal of Broadcasting and Electronic Media, Volume 57, Issue 3, pp. 409-426, 2013.

Hochman, Nadav and Lev Manovich. Visualizing Spatio-Temporal Social Patterns in Instagram Photos. In proceedings of the GeoHCI Workshop in conjunction with ACM CHI 2013. Paris, France, April 2013.

Hochman, Nadav and Manovich, Lev. Zooming into an Instagram City: Reading the local through social media. First Monday, 6/1/2013.

Adelheid Heftberger. "Das Potential der reduktionslosen Visualisierung am Beispiel von DAS ELFTE JAHR und DER MANN MIT DER KAMERA von Dziga Vertov." Chronos-Verlag, Geschichte und Informatik (forthcoming 2014).

Navas, Eduardo. Modular Complexity and Remix: The Collapse of Time and Space into Search. Published in AnthroVision 1.1 (2012); released on softwarestudies.com on 4/19/2013.

Manovich, Lev. Museum Without Walls, Art History without Names: Visualization Methods for Humanities and Media Studies. Oxford Handbook of Sound and Image in Digital Media, edited by Carol Vernallis, Amy Herzog, and John Richardson (Oxford University Press, 2013).

Manovich, Lev. Media Visualization: Visual Techniques for Exploring Large Media Collections. The International Encyclopedia of Media Studies, Volume VI: Media Studies Futures, ed. Kelly Gates, (Blackwell, 2013).

Manovich, Lev. Visualing Vertov. First part published in Russian Journal of Communication (Taylor & Francis); second part to appear in Cinematicity, eds. Jeff Geiger and Karin Littau (Edinburgh University Press). 1/11/2013.

Manovich, Lev. The Language of Media Software. 2/18/2013. Forthcoming in The App Book, eds. Svitlana Matviyenko and Paul Miller (The MIT Press, forthcoming 2014).

Manovich, Lev. Media After Software. Journal of Visual Culture. 4/5/2013.

Manovich, Lev. Visualizing Social Photography. 11/2013. Forthcoming in Aperture #214, winter 2014.

Manovich, Lev. Software is The Message. 12/2013. Forthcoming in Journal of Visual Culture, spring 2014.

Manovich, Lev. The Algorithms of Our Lives. The Chronicle of Higher Education, 12/16/2013.

Visualizando fotografias sociais

Lev Manovich, 2013.
Tradução: Cicero Inacio da Silva e Jane de Almeida


No prelo a ser publicado na revista Aperture #214 (2014). 


Neste verão o Museu de Arte Moderna de Nova Iorque (MoMA) pediu ao grupo de Estudos do Software, um programa que iniciei em 2007, para analisar como a visualização poderia ser usada como uma ferramenta de pesquisa e, talvez, como um meio para mostrar sua coleção de fotografias de maneira inovadora. Tivemos acesso a aproximadamente vinte mil fotografias digitalizadas, que combinamos em uma só imagem de super alta definição usando nosso software. Isso nos permitiu visualizar todas as imagens de uma só vez, rolando das imagens do início da mídia fotográfica até imagens registradas recentemente, abrangendo países, gêneros, técnicas e diversos aspectos da sensibilidade dos fotógrafos. Praticamente cada ícone fotográfico foi incluído - imagens que eu tinha visto repetidamente reproduzidas. Minha habilidade em facilmente poder realizar um zoom em cada imagem e estudar seus detalhes, ou diminuir o zoom para ver a imagem em sua totalidade, foi quase uma experiência religiosa. 

Observar vinte mil fotografias simultaneamente pode soar surpreendente, uma vez que até mesmo a maior galeria do museu possui, no máximo, uma centena de obras. E isso que a coleção do MoMA, usando os padrões do século XX, pode ser considerada escassa em comparação com os enormes repositórios de fotografias existentes em sites de compartilhamento de mídia, tais como Instagram, Flickr e 500px. (o Instagram sozinho já contém mais de um bilhão de fotografias, enquanto que os usuários do Facebook sobem mais de dez bilhões de imagens todos os meses). A ascenção da "fotografia social", lançada pelo Flickr em 2005, inaugurou novas possibilidades fascinantes para o campo da pesquisa cultural. A foto-universo criada por centenas de milhares de pessoas pode ser considerada como um mega-documentário, sem roteiro nem diretor, mas a escala desse documentário requer ferramentas computacionais - bases de dados, motores de busca e visualização, para que possamos assistí-lo.

Explorar (minerar) as partes constituintes desse "documentário" pode nos ensinar sobre a fotografia vernacular e hábitos que regem o desenvolvimento da imagem digital. Quando as pessoas se fotografam, será que elas privilegiam estilos de enquadramentos específicos, como os fotógrafos profissionais? Será que os turistas que visitam Nova Iorque fotografam os mesmos objetos; será que suas escolhas são culturalmente determinadas? E quando eles fotografam os mesmos objetos (por exemplo, plantas no High Line Park em Manhattan), eles usam as mesmas técnicas?

Para começar a responder a essas perguntas, podemos usar computadores para analisar os atributos visuais, os conteúdos de milhões de fotos e as descrições que as acompanham, tais como tags, coordenadas geográficas, data e hora dos uploads e, a partir daí, interpretar os resultados. Apesar dessa pesquisa só ter começado há poucos anos, já existe um número significativo de projetos interessantes que apontam para o futuro de uma "sociologia visual computacional" e de um "foto criticismo computacional".

Em 2009 David Crandall e seus colegas do Departamento de Ciência da Computação na Universidade de Cornell publicaram um artigo intitulado "Mapeando as Fotos do Mundo" (Mapping the World’s Photos), com base na análise de cerca de trinta e cinco milhões de fotografias do Flickr. Como parte da pesquisa, eles criaram um mapa constituído por todos os locais onde essas imagens foram tiradas (("world heat map"). As áreas com mais fotos parecem mais brilhantes , enquanto que aquelas com menos fotografias são mais escuras. Não é surpreendente que os Estados Unidos e a Europa Ocidental estão bem iluminados, enquanto que o resto do mundo permanece na escuridão, indicando uma cobertura mais esporádica. Mas o mapa também revela alguns padrões inesperados - as linhas costeiras da maioria dos continentes são muito brilhantes, enquanto os interiores dos continentes, com as notórias exceções ​​dos Estados Unidos e da Europa Ocidental, permanecem completamente no escuro.

Crandall e sua equipe, usando o conjunto de fotos coletadas, também determinaram os locais mais fotografados em vinte e cinco áreas metropolitanas. Isso os levou a novas descobertas - o quinto local mais fotografado de Nova York foi a loja da Apple no centro; A Tate Modern foi classificada como número dois em Londres.

O projeto de foto-mapeamento Locals and Tourists, criado em 2010 pelo artista de dados e desenvolvedor de software Eric Fisher, focou em uma pergunta, provavelmente motivada por tais informações: quantas dessas imagens foram capturadas por turistas ou moradores locais, e como que essa diferenciação pode revelar diferentes padrões? O projeto "Moradores e Turistas" de Fisher apontou os locais com um grande número de fotografias no Flickr e usou cores para indicar quem as tirou: as imagens azuis foram tiradas por habitantes locais e as imagens vermelhas foram registradas por turistas. As fotos com cores amarelas podem ter sido registradas por outros grupos. No total ele mapeou 136 cidades e depois compartilhou esses mapas no Flickr. Em seu mapa de Londres vemos como os turistas freqüentam alguns locais bem conhecidos, todos eles no centro de Londres, enquanto que os moradores locais cobrem toda a cidade, mas documentam imagens menos assiduamente.

Esses projetos pioneiros usam metadados para revelar padrões em fotografia social. Contudo, eles não usaram imagens reais em suas visualizações, uma prática explorada pela primeira vez, ao menos que eu saiba, pelo artista James Salavon. Para projetos como Every Playboy Centerfold, 1988–1997 e Homes for Sale, 1999-2002, Salavon fez uma composição de um número de imagens para revelar as convenções fotográficas utilizadas para representar assuntos específicos. Seu trabalho mais recente, Good and Evil '12, 2012, é composto por dois painéis, cada um mostrando cerca de vinte e cinco mil fotografias resultantes das buscas realizadas no Bing das cem palavras mais positivas ou negativas em Inglês.

Os mídia artistas como Salavon demonstram como a visualização pode revelar padrões no conteúdo de coleções de imagens de grandes dimensões. Em 2007 montei um laboratório de pesquisa para explorar ainda mais essas ideias e desenvolver ferramentas de visualização de código aberto que podem ser usadas por quem trabalha com historiadores da arte, imagens, estudiosos de cinema, mídia e curadores. Uma das nossas ferramentas de software pode analisar as propriedades visuais (como contraste, tons de cinza , textura, cores dominantes , orientações de linha) e algumas dimensões do conteúdo (presença e posição dos rostos e corpos) em qualquer número de imagens. Outra ferramenta pode utilizar os resultados desta análise para posicionar todas as imagens em uma única visualização de alta resolução ordenada por suas propriedades e metadados. Nós usamos essas ferramentas para visualizar uma variedade de coleções de imagens, que vão de cada uma das capas da revista Time entre 1923 e 2009, num total de 4.535 capas, até um milhão de páginas de mangá.

Em nosso projeto recente, Phototrails, o doutorando em História da Arte Nadav Hochman, o designer/ programador Jay Chow e eu começamos a explorar padrões em fotos enviadas para sites de mídia social. Na primeira fase do projeto, baixamos e analisamos 2,3 milhões de fotos do Instagram advindas de treze cidades globais. Uma das nossas visualizações mostra 53.498 fotos compartilhadas por pessoas em Tóquio durante alguns dias consecutivos. A progressão das atividades dominantes das pessoas ao longo de um dia de trabalho, jantando, saindo - reflete-se na mudança de cores e brilho das imagens. Nenhum dia é igual. Alguns são mais curtos do que os outros, ou a progressão entre as diferentes actividades é muito gradual, enquanto que em outros é mais nítida. Juntas, essas fotos criam um "documentário agregado" de Tokio - um retrato da mudança dos padrões temporais da cidade que foram reunidos a partir de milhares de atividades documentadas .

Mas os documentários a partir de um conjunto de imagens são algo novo? O documentário experimental Um Homem com uma câmera (1929) de Dziga Vertov, que retrata um dia na vida de uma cidade soviética, pode ser considerado um precursor dessa forma. O filme combina filmagens realizadas em três cidades ucranianas - Odessa, Khartiv e Kiev - ao longo de um período de três anos. Vertov queria comunicar ideias particulares, como a construção de uma sociedade comunista que orientou a seleção e edição de sua filmagem.

Nossas visualizações de hábitos humanos renderizados por meio de fotografias advindas do Instagram não refletem somente um único ponto de vista. Mesmo assim, elas são tão subjetivas quanto as mais tradicionais fotografias. Assim como um fotógrafo decide sobre o enquadramento e a perspectiva, nós tomamos decisões formais sobre como mapear as imagens, organizá-las por datas de upload, ou cor, ou brilho e assim por diante. Mas, visualizando o mesmo conjunto de imagens de várias maneiras (veja aqui um exemplo que usa uma coleção de obras de arte de Mark Rothko), lembramos que nenhuma visualização oferece uma interpretação objetiva, assim como nenhuma imagem documental única e tradicional jamais poderia ser considerada neutra,  ao invés disso, a diversidade das fotografias do Instagram destacam a variedade de padrões complexos de vida que se desdobram em cidades que nunca podem ser de forma total visualmente capturadas em uma única visualização, apesar da nossa capacidade de usar milhões de fotos do Instagram.

( Novembro, 2013)

"Software is the Message" - new mini article (1000 words) from Lev Manovich

Software is the Message

Lev Manovich. 2013.

Forthcoming in Journal of Visual Culture, special issue "Marshall McLuhan's Understanding Media: The Extensions of Man @ 50" (Spring 2014).

(Note: Some parts of this text come from my book Software Takes Command, Bloomsbury Academic, 2013.)



Did Marshall McLuhan “miss” computers? In his major work, Understanding Media: The Extensions of Man (1964) the word “computer” appears 21 times in the book, and a few of those references are to “computer age.” However, despite these references, his awareness of computers did not have significant effect on his thinking. The book contains two dozens chapters each devoted to a particular medium—which for McLuhan range from writing and roads to cars and television. (The last chapter “Automation” addresses the role of computers for industrial control, but not its other roles).

The reasons for this omission are not hard to understand. McLuhan’s theories were focused on the media that were widely employed by regular people in human history. In 1964, the popular media for representation and communication did not yet include computers. Although by the end of the 1960s computer systems for design, drawing, animation, word processing were also developed (along with the first compute network that eventually became the Internet), these systems were only used by small communities of scientists and professionals. Only after the introduction of a PC in 1981, these inventions started to be disseminated to the masses.

As a result, software has emerged as the main new media form of out time. (I say “software” rather than “digital computers” because the latter are used to do everything in our society, and often their use does not involve software visible to the ordinary users – like the systems inside a car.) Outside of certain cultural areas such as crafts and fine art, software has replaced a diverse array of physical, mechanical, and electronic technologies used before the twenty-first century to create, store, distribute and access cultural artifacts, and communicate with other people. When you write an article in Word, you are using software. When you are composing a blog post in Blogger or WordPress, you are using software. When you tweet, post messages on Facebook, search through billions of videos on YouTube, or read texts on Scribd, you are using software (specifically, its category referred to as “web applications” or “webware”—software which is accessed via web browsers and which resides on the servers).

McLuhan’s theories covered the key “new media” of his time – television, newspapers and magazines with color photos, advertising, and cinema. Just as these mediums, software medium took decades to develop and mature to the point where it dominates our cultural landscape. How does the use of professional media authoring applications influences contemporary visual imagination? How does the software offered by social media services such as Instagram shapes the images people capture and share? How do particular algorithms used by Facebook to decide what updates from our friends to show in our News Feed shape how we understand the world? More generally, what does it mean to live in “software society”?

In 2002, I was in Cologne, Germany, and I went into the best bookstore in the city devoted to humanities and arts titles. Its new media section contained hundreds of titles. However, not a single book was devoted to the key driver of the “computer age”: software. I started going through indexes of book after book: None of them had the word “software” either. How was that possible? Today, thanks to efforts of my colleagues in the new academic field of “software studies,” the situation is gradually improving. However, when I looked at indexes of works of key media theorists of our time published in the last year, I still did not find entry for “software.” Software as a theoretical category is still invisible to most academics, artists, and cultural professionals interested in IT and its cultural and social effects.

Software is the interface to our imagination and the world—a universal language through which the world speaks, and a universal engine on which the world runs. Another term that we can use in thinking about software is that of a dimension (think of three dimensions that we use to define space). We can say that at the end of the twentieth century humans have added a fundamentally new dimension to everything that counts as “culture — that of software.

Why this conceptualization is useful? “Cultural software” is not simply a new object—no matter how large and important—which has been dropped into the space which we call “culture.” And while we can certainly study “the culture of software”—programming practices, values and ideologies of programmers and software companies, the cultures of Silicon Valley and Bangalore, etc.—if we only do this, we will miss the real importance of software. Like the alphabet, mathematics, printing press, combustion engine, electricity, and integrated circuits, software re-adjusts and re-shapes everything it is applied to—or at least, it has a potential to do this. Just as adding a new dimension adds a new coordinate to every point in space, “adding” software to culture changes the identity of everything that a culture is made from. In this respect, software is a perfect example of what McLuhan meant when he wrote that the “message of any medium or technology is the change of scale or pace or pattern that it introduces into human affairs.”

However, the development and current hegemony of software does not simply perfectly illustrate the points McLuhan made 50 years ago. It also challenges these ideas. Here is how.

In the first few decades, writing new software was the domain of professionals. However, already in the 1960s Ted Nelson and Alan Kay proposed that computers could become a new kind of cultural medium. In their paradigm, the designers would create programming tools, and the users would invent new media using these tools. According, Alan Kay called computers the first metamedium whose content is “a wide range of already-existing and not-yet-invented media.”

This paradigm had far-reaching consequences for how software medium functions today. Once computers and programming were democratized enough, some of the most creative people of our time started to focus on creating these new structures and techniques rather than using the existing ones to make “content.” During the 2000, extending the computer metamedium by writing new software, plugins, programming libraries and other tools became the new cutting-edge type of cultural activity.

For example, GitHub, a popular platform for sharing and developing open source tools, houses hundreds of thousands of software projects. Making new software tools is central for the fields of digital humanities and software art. And certainly, the key “media companies” of our time such as Google, Facebook, or Instagram do not create content. Instead they constantly refine and expand their software tools used by hundreds of millions of people to make content and to communicate.

Thus, its time to update Understanding Media. It is no longer the medium that is the message today. Instead, the software is the message. And continuously expanding what humans can express and how they can communicate is now our “content.”