Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
539 views
in Technique[技术] by (71.8m points)

python - SQLAlchemy and UnicodeDecodeError

I'm getting

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 0: ordinal not in range(128)

when I pass text coming from a MySQL database, which I am accessing using SQLAlchemy, to this function:

re.compile(ur"<([^>]+)>", flags=re.UNICODE).sub(u" ", s)

The database encoding is utf-8 and I am even passing the encoding to the create_engine function of SQLAlchemy.

Edit: This is how I am querying the database:

doc = session.query(Document).get(doc_id)
s = doc.title

By suggestion, I passed s.decode('utf-8') to sub . The error above disappeared, but I get a different error for a different document:

UnicodeDecodeError: 'utf8' codec can't decode byte 0xeb in position 449: invalid continuation byte

The database table is defined like this:

CREATE TABLE `articles` (
  `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
  `title` varchar(255) DEFAULT NULL,
  `cdate` datetime DEFAULT NULL,
  `link` varchar(255) DEFAULT NULL,
  `content` text,
  UNIQUE KEY `id` (`id`),
  UNIQUE KEY `link_idx` (`link`)
) ENGINE=InnoDB AUTO_INCREMENT=4127834 DEFAULT CHARSET=utf8;

Any help would be greatly appreciated

See Question&Answers more detail:os

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Reply

0 votes
by (71.8m points)

I have solved the issue. The title column was being returned by SQLAlchemy as a str and not Unicode. I thought adding encoding='utf8' as an argument to create_engine would take care of this, however, the right way to do it is to pass it in the database URI: mysql://me@myserver/mydatabase?charset=utf8 .

Thank you for all your answers!


与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
OGeek|极客中国-欢迎来到极客的世界,一个免费开放的程序员编程交流平台!开放,进步,分享!让技术改变生活,让极客改变未来! Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

...